
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2010-09 

Naval Fuel Management System (NFMS) a 
decision support system for a limited resource 


Conner, R. Brian. 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/5166 


This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Callwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional publicatkios created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. Caftiouo, NPS's first 
appointed — and published — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 Univefsity Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


THESIS 


RECOMMENDATIONS TO COMMANDER, NAVAL 
SURFACE FORCES, FOR ACHIEVING A STRATEGIC 
APPROACH TO THE ACQUISITION OF INFORMATION 
SYSTEMS AND SERVICES 

by 

Melissa L. Williams 
Robert Brian Conner 

September 2010 

Thesis Advisor: Glenn Cook 

Second Reader: Brad Naegle 


Approved for public release; distribution is unlimited 




THIS PAGE INTENTIONALLY LEET BLANK 



REPORT DOCUMENTATION PAGE 


FormApprovedOMBNa 0704-0188^ 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, 
searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send 
comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to 
Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 
22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

I. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 

September 2010 Master’s Thesis 

4. TITLE AND SUBTITLE Recommendations to Commander, Naval Surface 5. EUNDING NUMBERS 

Forces, for Achieving a Strategic Approach to the Acquisition of Information 
Systems and Services 

6. author" 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PEREORMING ORGANIZATION 

Naval Postgraduate School REPORT NUMBER 

Monterey, CA 93943-5000 

9. SPONSORING /MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING 
N/A AGENCY REPORT NUMBER 

II. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy 

or position of the Department of Defense or the U.S. government. IRB Protocol number_N/A_. 

12a. DISTRIBUTION / AVAII.ABIITTY STATEMENT I2b. DISTRIBUTION CODE 

Approved for public release; distribution is unlimited 

13. ABSTRACT (maximum 200 words) 

Historically, Commander, Naval Surface Forces (CNSF), has awarded multiple service contracts ranging between $7 
million to $8 million dollars, which have resulted in outflow of funds to maintain disparate systems that produce non- 
integrated data structures. The command requires a method to maintain or improve their capability while reducing 
their operating costs. The contracts meet CNSF major automaton needs for training, Websites, and reporting tools. 
However, CNSF desires an enterprise level integration plan for acquiring information systems (IS), and an 
architecture plan that will align all systems and metrics. This change will increase efficiency, reduce redundancy, and 
establish an enterprise framework. CNSF traditional decision making has resulted in several disparate programs that 
require separate contracts and limited integration. This structure has fueled excessive service fees, stovepipe systems, 
and the organization is stuck paying to maintain the operation. This study will assist in identifying the current 
traditional behaviors and introduce a more strategic approach to decision making. Additionally, our goal is to evolve 
the current IS acquisition decision making from its current state to one of a more strategic-based concept The focus of 
this thesis is on reengineering the CNSF decision-making process for acquiring IS capabilities. 

14. SUBJECT TERMS Type Keywords Here Acquisition Strategic Characteristics, Acquisition 15. NUMBER OE 
Strategy, Performance Based Specifications, Evolutionary Acquisition, Product Line Architecture, PAGES 

Command Naval Surface Forces, Surface Warfare Enterprise, Naval Aviation Enterprise, GAO Case 177 


Study, Strategic IS Vision, Kruskal- Wallis Test, Wilcoxon- Rank Sum Test, Spearman Correlation 
Test, Acquisition Implementation Steps 

16. PRICE CODE 

17. SECURITY 

CLASSIEICATION OE 

REPORT 

Unclassified 

18. SECURITY 

CLASSIEICATION OE THIS 
PAGE 

Unclassified 

19. SECURITY 

CLASSIEICATION OE 
ABSTRACT 

Unclassified 

20. LIMITATION OE 
ABSTRACT 

UU 


NSN 7540-01-280-5500 Standard Form 298 (Rev. 8-98) 


Prescribed by ANSI Std. Z39.18 


1 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release; distribution is unlimited 


RECOMMENDATIONS TO COMMANDER, NAVAL SURFACE FORCES FOR 
ACHIEVING A STRATEGIC APPROACH TO THE ACQUISITION OF 
INFORMATION SYSTEMS AND SERVICES 


Melissa L. Williams 
Lieutenant Colonel, United States Army 
B.S., University of South Carolina, 1992 

R. Brian Conner 
Lieutenant, United States Navy 
BA., Old Dominion University, 2004 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN INFORMATION TECHNOLOGY MANAGEMENT 

from the 

NAVAL POSTGRADUATE SCHOOL 
September 2010 


Author: Melissa L. Williams 

R. Brian Conner 


Approved by: Glenn Cook 

Thesis Advisor 


Brad Naegle 
Second Reader 


Dan C. Boger 

Chairman, Department of Information Sciences 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


Historically, Commander, Naval Surface Forces (CNSF), has awarded multiple service 
contracts ranging between $7 million to $8 million, which have resulted in outflow of 
funds to maintain disparate systems that produce nonintegrated data structures. The 
command requires a method to maintain or improve its capability while reducing its 
operating costs. The contracts meet CNSF major automaton needs for training. Websites, 
and reporting tools. However, CNSF desires an enterprise level integration plan for 
acquiring information systems (IS), and an architecture plan that will align all systems 
and metrics. This change will increase efficiency, reduce redundancy, and establish an 
enterprise framework. CNSF traditional decision making has resulted in several disparate 
programs that require separate contracts and limited integration. This structure has fueled 
excessive service fees and stovepipe systems, and the organization is stuck paying to 
maintain the operation. This study will assist in identifying the current traditional 
behaviors and introduce a more strategic approach to decision making. Additionally, our 
goal is to evolve the current IS acquisition decision making from its current state to one 
of a more strategic-based concept. The focus of this thesis is on re-engineering the CNSF 
decision-making process for acquiring IS capabilities. 
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I. INTRODUCTION 

A. ACQUISITION STRATEGY FOR PROCUREMENT OF INFORMATION 

SYSTEMS AND SERVICES TO MEET A PRESCRIBED IS STRATEGY 

Information Systems (IS) provide vital enhancement to the mission capability of 
both operational and supporting commands within the DoD. As the mission requirements 
of these commands evolve, their IS needs also change. The dilemma for these 
commands, their Military Departments (MILDEPS), and the DoD is the acquisition of IS 
that leverage newer dynamic technologies. These systems must meet both mission and 
strategic IS goals, and fit within the Department’s vision of an enterprise architecture 
framework. 

Over the last decade, services have become increasingly reliant upon IS, resulting 
in greater scrutiny on the acquisition process. The Clinger-Cohen Act of 1996 provides 
the documented framework that catalyzed review and reassessment of the government’s 
acquisition process as it relates to IS. The document sets forth requirements that should 
be met to achieve “maximizing the value and managing the risks of IS acquisitions 
(Clinger, 1996).” In order to meet these goals, a strategic approach to the acquisition of 
IT is necessary. The Acquisition Strategy Guide states: “An acquisition strategy achieves 
program objectives using a technical management approach. It is a framework for 
achieving core objectives of a program.” The guide provides a roadmap for meeting the 
goals of the Clinger-Cohen Act. 

B. BACKGROUND 

Historically, CNSF has awarded multiple IS service contracts ranging between 
$7 million and $8 million, which have resulted in outflow of funds to maintain disparate 
systems that produce non-integrated data structures. The command requires a method to 
maintain or improve their capability while reducing their operating costs. The contracts 
meet CNSF major automaton needs for training, Web sites, and reporting tools. However, 
CNSF desires an enterprise level integration plan for acquiring IT systems, and an 
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architecture plan that will align all systems and metrics. This change will increase 
efficiency, reduce redundancy, and establish an enterprise framework. 

CNSF traditional decision making has led to disjointed programs that require 
separate contracts. This structure has fueled excessive service fees, stovepipe systems, 
and the organization is stuck paying to maintain the operation. This study will help 
identify the current traditional behaviors and introduce a more strategic approach to 
decision making. Additionally, our goal is to evolve the current acquisition plan from its 
current state to one of a more strategic-based concept (Commander, Naval Surface 
Forces, 2005). 

The focus of this thesis is on reengineering the Commander, Naval Surface Forces 
(CNSF) decision-making process for acquiring IS capabilities. This process warrants 
reengineering in order for it to become compliant with DODI 5000.02, which advocates 
an enterprise-wide approach for both planning and execution of IS acquisition (DODI 
5000.02, 2008) 

These systems support the administrative mission of the U.S. Navy Commander 
and the fleet of assets it supports. The fleet has become completely reliant upon 
information systems for administrative control. Their criticality to the evolving missions 
of the Navy requires an acquisition process that can provide system integration, rapid 
fielding, and enterprise scalability. 

C. RESEARCH QUESTIONS 

The following research questions were formulated to assist in developing a 
strategic acquisition process to procure information systems and meet CNSF strategic IS 
requirements. 

1. Primary Question 

Which Strategic Approach to IS acquisitions broadly 
supports both CNSF missions and business practices? 

2. Secondary Questions 

What are the different IS Acquisition approaches? 
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What types of IS Acquisition approaches do other 
organizations with a similar structure use? 

How do similar organizations measure ROI in their 
acquisition programs? 

D. POTENTIAL BENEFITS 

This thesis is an externally sponsored project for CNSF. It is the third part of an 
overall three-part project. CNSF has coordinated with three sets of students to provide 
the following: 

1) Assessment of contracting measures 

2) Enterprise IS strategy 

3) Enterprise plan for acquiring service contracts 

This thesis provides recommendations for IS acquisition. It will feed into the 
development of an IS acquisition strategy, based on the recommended IS Strategy from 
Group two in the project. It will also provide a means for establishing and controlling IS 
capabilities with minimum impact on the organizational structure. 

E. METHODOLOGY 

The research methodology for thesis entailed data collection, examination and 
analysis from policy and guidance documentation from DoD and DON. Additionally, 
theoretical and commercial acquisition strategies were examined analysis and relevance. 
Findings from the preceding groups involved in this project was reviewed and referenced 
to identify basic requirements and baseline spending on IS strategies. Discussion via e- 
mail and telephone were utilized gather information about CNSF current IS acquisition 
processes. Conclusions were drawn and recommendations were provided to CNSF as to 
an appropriate acquisition strategy to meet current and future IS needs. 

F. ORGANIZATION OF STUDY 

Chapter II Overview of Current Acquisition Strategies 

Chapter II will include academic review of current government and commercial 

acquisition requirements and strategies that have been commonly used are widely 
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successful. The research will concentrate on how these strategies can assist CNSF in 
achieving successful IS acquisitions, and identify shortfalls in CNSF current acquisitions 
process. The literature review will focus primarily on acquisition strategies and 
acquisition management advantages to improve CNSF current and future IS acquisition 
programs. 

Chapter III Background of Commander, Naval Surface Forces (CNSF), 

Organization and Approach to IS Acquisition 

This chapter will examine CNSF acquisition processes and determine advantages 
and disadvantages of their current methods used in developing Surface Warfare 
Enterprise (SWE) applications. We will provide history and background of the CNSE 
organization, the SWE and Commander, Naval Air Eorces (CNAE). The background 
information discussed will be used to compare and contrast organizational norms and 
behaviors with CNAE and similar commercial organizations in order to identify 
advantages and disadvantages of CNSE processes. We conducted interviews with 
contracting and development staff personnel to determine the acquisition roadmap used 
by CNSE to include research, development, contracting, and procurement of the SWE 
applications from requirements to delivery. 

Chapters IV and V Analysis, Conclusions and Recommendations 

After acquisition strategies have been examined and analyzed, and CNSE 
acquisition processes have been reviewed recommendations will be made for a more 
robust and effective acquisition strategy for developing systems that will meet strategic 
IS requirements and fit within the enterprise needs of the CNSE. Chapter IV will 
highlight solutions to mitigate the disadvantages of CNSE current acquisition processes 
discussed in chapter three and explain recommendations for successful execution of an 
acquisition strategy that will be more suitable for CNSE business process and IS needs. 
Chapter five will summarize this research and conclusions, and give recommendations 
for future work. 
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II. A STRATEGIC APPROACH TO SYSTEM ACQUISITION 


A. DEFINITION OF ACQUISITION STRATEGY 

An acquisition strategy (AS) achieves program objectives using a technical 
management approach. It is a framework for achieving core objectives of a program, 
which are: planning, organizing, staffing, controlling, and leading a program (Acquisition 
Strategy Guide, 2005). Strategically approaching the acquisition process organizes 
priorities that meet specific business objectives and Information System (IS) 
requirements. An effective strategy balances cost and effectiveness through the 
following measures: development of technological options, exploration of design 
concepts, and acquisition activities (Acquisition Strategy Guide, 2005). An AS is of 
great importance to those organizations that primarily acquire rather than develop, this 
practice area is especially important for U.S. government agencies, such as the U.S. 
Department of Defense (DoD) (SEI, 2010). DoD Instruction 5000.02 mandates the 
inclusion of an AS in the acquisition. The Program Manager (PM) shall prepare, and the 
Milestone Decision Authority (MDA) shall approve an Acquisition Strategy to guide 
activity during Engineering Manufacturing and Development (EMD). 

The AS shall describe how the PM plans to employ contract incentives to achieve 
required cost, schedule, and performance outcomes (Department of Defense, 2008). 
Eigure 1, from the Acquisition Strategy Guide, displays the milestones and decision 
points that are integral to the acquisition process. 
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Figure 1. Defense Acquisition System with milestones and decision points (From 

Defense Acquisition University, 2003) 


B. ELEMENTS IN ACQUISITION STRATEGY DEVELOPMENT 

An effective acquisition strategy is a key requirement of both system procurement 
and overall acquisition. A primary goal in developing an acquisition strategy is the 
minimization of the time and cost of satisfying an identified, validated need consistent 
with common sense and sound business practices (Defense Acquisition University, 
2003). It should cover system initial development through end-of-life system disposal. 
The effectiveness of an AS is based upon five characteristics known as Acquisition 
Strategy Characteristics (ASC), which are: 

• Realism 

• Stability 

• Resource balance 

• Flexibility 

• Managed risk 

I. Acquisition Strategy Characteristics (ASC) 

These characteristics assist in giving the AS credibility and facilitate buy-in of 
key stakeholders and decision makers. The goals of any acquisition must be realistic and 
attainable, and that milestones can be achieved within the allotted timeframe of the 
program. Acquisition stability is the characteristic that inhibits negative external or 
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internal influences from seriously disrupting program progress (Defense Systems 
Management College, 1999). Changes in cost, resources, timeline, etc., can have far 
reaching effects into the overall acquisition of a system. 

The greater the stability of a program, the more likely it is to withstand these 
effects and remain on target, and below estimated cost. Changes in requirements, 
funding, personnel or organization can all negatively impact acquisition stability and 
result in program disruption. However, a sense of purpose, top-level support, and firm 
commitment can contribute to a program’s overall stability and resistance to the 
aforementioned influences. 

Resource balance is a condition of equilibrium between and within major program 
objectives that are competing for resources. The achievement of cost, schedule, and 
performance requirements uses resources of time, people, facilities, and money—all of 
which are limited (Defense Systems Management College, 1999). Competition for 
funding is one of the biggest influences affecting programs. As a resource, adequate 
funding is vital to the survival and implementation of any acquisition. Issues with 
resource balance most commonly occur when there is a conflict between user 
requirements and funding constraints. To meet congressional intent to control costs, the 
Program Manager (PM) is forced to balance the user’s desire for the system that will best 
perform and meet his needs, and the desire of the organizational Comptroller to keep 
spending at or below budget constraints. 

Flexibility is a characteristic of the acquisition strategy related to the ease with 
which changes can be accommodated without significant changes in resource 
requirements (Defense Systems Management College, 1999). An acquisition strategy 
must allow for changes in requirements, funding, contracts, and design, in order facilitate 
program growth and development, satisfy funding constraints and best meet user needs. 
Flexibility is possibly the most important AS characteristic. Flexibility must be built into 
every aspect of the AS. 

The acquisition of any program will encounter uncertainty. These uncertainties 
represent risk, which could negatively impact the overall acquisition. Managing this risk 
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is necessary throughout the acquisition process and should be included in the AS. Risk 
management is concerned with the identification, assessment, mitigation, and handling of 
uncertainties that threaten cost, schedule, and performance objectives (Defense 
Acquisition University, 2003). The AS must address the dynamic sources of risk 
associated with any program. Therefore, the AS must be continually monitored and 
updated to address changes and mitigate risks. 

2. Key Items for an Acquisition Strategy 

In order to mitigate risk and provide for greater stability within the AS, funding 
sources and goals must be outlined and considered. The strategy should discuss how the 
funding lines will affect the chosen acquisition approach, either (i.e., evolutionary, single- 
step). The issue of cost is a common issue that budgeting professionals must tackle in the 
decision to acquire new systems. Program costs must be adequately estimated, and 
aggressively managed to ensure that system performance is met, and costs do not exceed 
funding constraints. The Concept of Cost As an Independent Variable (CAIV) should be 
used in the acquisition strategy as a means of address [ing] methodologies to acquire and 
operate affordable DoD systems by setting aggressive, achievable cost objectives and 
managing achievement of these objectives (Defense Systems Management College, 
1999). 


a. Cost as an Independent Variable (CAIV) 

Cost as an Independent Variable integrates best practices shown to be 
effective within the commercial sector with government acquisition initiatives in order to 
provide DoD entities with superior war fighting assets and capabilities. It is executed by 
utilizing the three measures by which a program’s success is often measured, which are: 
cost, schedule, and system performance. Under this concept, performance and schedule 
are allowed to vary within the program while cost is held as independent. Once the 
required capability and design have been agreed upon, a cost constraint is formulated and 
the system is then constrained by this cost and the capability is the overall end state. 
CAIV should be instituted early in the acquisition process. It is important to note that 
this cost is considered over the lifecycle of the system, and incorporates total cost 
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ownership. CAIV is an acquisition philosophy that emphasizes keeping system life cycle 
cost within an established range by trading the other system acquisition variables of 
performance or schedule (Defense Acquisition University, 2010). 

An example of CAIV is the Abrams Tank. The Army is considering a 
replacement for the Abrams tank. Traditional cost analysis would indicate that the cost 
of the new tank (development, production, sustainment) would be a function or other 
variables, such as the tank’s combat weight, armament, speed, survivability, 
maintainability, etc. In other words. Cost = f (weight, armament...). Therefore, cost 
depends on the set levels of the other variables. CAIV reverses this role. Cost is 
independently set to some budgeted value, and a determination is made on how much 
performance can be derived from that cost (Defense Acquisition University, 2010). 

C. DESCRIPTION OF STRATEGIC ACQUISITION APPROACHES 

There are several acquisition strategies for the various goods and/or services that 
must be procured to meet operational objectives. It is important to design a strategy that 
best fits the organization’s business needs, processes, culture, needs of the project and 
environment. A good acquisition strategy is realistically tailored to program objectives 
and constraints, and is flexible enough to allow innovation and modification as the 
program evolves (Guide, 2005). In the case of Information Systems (IS), the process 
must be agile and flexible enough to rapidly deliver meaningful incremental capability 
(Defense Science Board, 2008). A clear, credible acquisition strategy contains the 
following characteristics: realism, stability, resource balance, flexibility, and managed 
risk (Guide 2005). These characteristics aid in meeting the objectives of the strategy. 
Some acquisition strategies are: 

• Evolutionary Acquisition 

• Product Line Acquisition 

• Performance-Based Acquisition 
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1. 


Evolutionary Acquisition Strategy (EAS) 


According to ACQuipedia, EAS is defined as the preferred DoD strategy for rapid 
acquisition of mature technology that delivers capability in militarily useful increments 
with a recognized need for upfront future capability improvements. The increments or 
steps used in EAS help to refine already well-defined requirements. Each EA program 
increment is treated in theory as if it were a separate program with its own unique 
developmental and procurement phases. 



Eigure 2. Evolutionary Acquisition capability gained with reference to time (Erom 

Defense Acquisition University, 2003) 

A series of less demanding, phased capability steps are identified and the 
available technologies are matched with more realistic requirements and capability 
expectations over time, which results in multiple steps of increasing capability available 
much sooner to the war fighter (RAND 2002). Eigure 2 displays the relationship between 
time and capability in the EAS process. Requirements become more refined, while 
capability and technology increase with each iteration. An evolutionary approach to 
acquisition requires considerable interaction between users and developers. In the EAS, 
development is executed incrementally. 
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a. 


Spiral Development 


Under spiral development the desired capability or mission need is 
identified at the outset, but the ultimate state of the system is still unknown. This type of 
development incrementally delivers the user the most capable iteration to meet refined 
requirements. Technology maturation and user feedback recursively effect future 
increments delivered through form of development. Spiral development is no longer 
used within the DoD. However, the explanation provided here is relative to a researched 
system that will be introduced and discussed in later chapters. 

b. Incremental Development 

In incremental development, a desired capability is identified, an end-state 
requirement is known, and that requirement is met over time by development of several 
increments, each dependent on available mature technology (Defense Acquisition 
University, 2003). EAS is a widely used method for acquiring systems. It is commonly 
used in the acquisition of large weapon systems that must be fielded rapidly to meet 
operational needs. Often, the end-state of the system is unknown. The system is 
dynamic, constantly evolving to meet changes in requirements. Figure 3, from the 
Defense Acquisition University, displays how incremental and spiral development 
interact under the evolutionary acquisition strategy. 
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Figure 3. Spiral and Incremental Development in EAS (From Defense Acquisition 

University, 2003) 


Operationally, there are challenges to implementing EAS into programs. 
Cost estimation at the initial phases of the program is one of the greatest concerns 
(RAND, 2005). However, throughout the process, costing is a continual challenge. In 
the case of IT, the cost to upgrade later increments, changes in technological maturity, 
etc., makes cost estimations difficult for program budgeting. Some benefits of EAS are: 

• Rapid deployment of an operational capability 

• Increments of the overall program are more easily managed 

• Enhanced granularity of requirements 

• Enhanced management of timeline 

• Allows for greater requirements flexibility 

The benefits and disadvantages of EAS should be considered for software 
intensive systems in which requirements are dynamic yet well defined and the overall 
end-state of the system is unknown. Although there are several advantages that an 
organization may leverage from an Evolutionary Acquisition Strategy, there are 
drawbacks to this approach. The below list is not all encompassing but merely represents 
a few of these drawbacks. The disadvantages of EAS are: 
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• Increased number of Milestone reviews 

• Difficulty of configuration management 

• Likelihood that the "User" representative will disagree to anything less 
than the objective solution 

• Operational issues related to differences in system capability 

• Cost increases may lead to breaches 

• Problems with programming and budgeting funding streams for 
multiple iterations. 

• Testing issues (Boudreau, 2010) 

An Evolutionary Acquisition Strategy is the preferred method of IS 
procurement within the DoD. This acquisition approach calls for either an incremental 
development. This approach allows for an initial operating capability (IOC) to be 
achieved as an initial solution to user requirements. It also provides greater flexibility for 
evolving mission needs. However, if initial costing estimations are not carefully 
executed, this approach can result in cost overruns. 

2. Product Line Acquisition (PLA) 

A product line acquisition (PLA) strategy is a plan of action for achieving a 
specific product line goal or result through contracting for products and service. Potential 
software services include elements of core asset development, product development, and 
management. Acquiring services means contractually engaging an identifiable task rather 
than furnishing an end product. A product line approach is a natural fit for specifying 
and coordinating efforts across a distributed or geographically dispersed workforce (SEI, 
2010 ). 

A product line is defined to be a group of products sharing a common, managed 
set of features that satisfy specific needs of a selected market or mission (Bergey, 1998). 
PLA involves core asset development and product development using the core assets, 
both under the aegis of technical and organizational management (SEI, 2010). Eigure 4 
from Carnegie Mellon’s Software Engineering Institute demonstrates how the three 
essential activities interact in order to field product line architecture. Each rotating circle 
represents one of the essential activities. All three are linked together and in perpetual 
motion, showing that they are all essential, inextricably linked, and highly iterative, and 
can occur in any order (SEI, 2010). 
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Figure 4. Activity Interaction for PLA (From Carnegie Mellon University Software 

Engineering Institute, 2010) 

When considering using PLA, the architecture needs to be developed from the 
outset. One acquisition strategy, which is most applicable to government agencies that 
rely very heavily on acquisition, involves procuring only an architecture in the first stage, 
procuring other core assets in the second stage, and procuring products (built from the 
core asset base) in the third stage (SEI, 2010). There are four acquisition strategies for 
acquiring an architecture: 

• Systems Architect 

• Single Contractor 

• Collaborating Contractors 

• Standards Group 

a. Systems Architect 

In this architecture acquisition strategy, a single contractor develops the 
architecture for the system. However, the funding and ownership is attributed to the 
acquiring organization. A different architect is hired for the actual implementation of the 
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architecture. This is done to manage the risk of the architecture not meeting the program 
requirements. It is important to note that cost adherence could still be an issue in this 
strategy (SEI, 2010). 

b. Single Contractor 

In this strategy a single contractor develops some components or the 
system under the contract. The contractor supplies the architecture and ownership. 
Control over the system is negotiated between the contractor and the acquiring 
organization. 


c. Collaborating Contractors 

A contract is developed that requires a group of contractors to collaborate 
on developing an architecture that they all can use later. In addition, each of the 
contractors is awarded a contract to develop and maintain some of the system's 
components. Ownership of the architecture is usually shared among the development 
contractors, with the acquiring organization holding the licensing rights. The acquiring 
organization funds joint development and manages the architecture requirements (SEI, 
2010 ). 


d. Standards Group 

An architecture is either built by a “standards group” or conforms to 
established standards is acquired. Industry and/or government collaboration creastes a 
public architecture. The acquiring organization influences but does not control or own 
the product line architecture (SEI, 2010). 

Software Reuse is a critical part of the Product Eine concept. Software 
reuse is the process of creating software systems from existing software rather than 
building software systems from scratch (Krueger, 1992). In reuse proven technologies, 
standards and code are reused in new and subsequent products. The benefits of this 
approach are cost-savings, reliability, and efficiency. 
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For the military, product lines may leverage software and systems that 
have already been proven in the commercial sector to meet mission requirements, and/or 
core business processes. Pursuing a PLA can result in substantial cost savings and 
systems that are cohesive in nature. This cohesiveness can be helpful when integrating 
systems into a standardized architecture. Additionally, commands may use Commercial- 
of-the-Shelf (COTS) products, that interface well together, and can be modularly moved, 
removed, and upgraded when necessary. 

Operationally, we can find an example of Product Lines in ship classes. 
The Guided Missile Destroyer (DDG) is a standard system of systems that leverages 
proven technologies that have been iteratively developed to yield a line of products. The 
Tomahawk missiles that are a part of its armament are a result of software that can be 
used on other tactical platforms as well as the DDG. The Shipboard systems from the 
Combat Systems Suite to the Engineering Plant are a result of product line architecture 
acquisition. 

Product Line Architectures are characterized by the capability of being 
highly cohesive, yet loosely coupled. Apple’s family of products is a highly visible 
example of a PLA that is highly cohesive yet loosely coupled. A system that is highly 
cohesive means that its components fit well together in an almost seamless integration. 
In the case of ITunes, this software works with both Macintosh hardware and software 
systems (its native platform), and PC hardware and software systems, with virtually no 
integration issues. Systems with components that are loosely coupled can be used on 
multiple software or hardware instances with no issue to the stability or functionality of 
the system. Lor example, ITunes does not need an IPod to function. If ITunes is 
degraded the iPod will still function as configured. 
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e. Benefits of PLA 

Organizations can gain an advantage in their business mission area, 
through the implementation of PLA. By assessing and developing core assets and 
processes, and developing other systems and processes to reuse those core assets, an 
organization can derive numerous savings. Some of the organizational benefits of PLA 
are: 

• Increased product quality 

• Increased customer satisfaction 

• More efficient use of human resources 

• Decreased product risk 

The advantages gained in instituting Product Lines permit noticeable 
organizational improvements, which ultimately result in competitive advantage. 

f Disadvantages of PLA 

Although PLA can lead to numerous gains within an organization’s core 
processes, instituting this strategy can be somewhat difficult. It takes a certain degree of 
maturity in the developing organization to field a product line successfully. Technology 
change is not the only barrier to successful product line adoption. Changes in 
management and organizational practices are also involved. Successful adoption of 
software product line practice is a careful blend of technological, process, organizational, 
and business improvements (SEI, 2010). 

3. Performance-Based Acquisition 

Performance-Based Acquisition (PBA) is the government’s preferred approach 
for acquiring service (Denett, 2007). PBA involves acquisition strategies, methods, and 
techniques that describe and communicate measurable outcomes rather than direct 
performance processes. PBA is a technique for structuring all aspects of an acquisition 
around the purpose and outcome desired as opposed to the process by which the work is 
to be performed (government Services Administration, 2010). Performance-Based 
Service Acquisition (PBSA) strategies strive to adopt the best commercial practices and 
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provide the means to reach world-class commercial suppliers, gain greater access to 
technological innovations, maximize competition and obtain the best value to achieve 
greater savings and efficiencies (Office of the Undersecretary for Acquisition 
Technology and Logistics, 2001). PBSA operates under the need to meet mission 
requirements as they arise. The performance of systems and services as well as 
contractors are measured and assessed to ensure outcomes are meeting mission need as 
specified. PBSA assists agencies in achieving the following objectives: 

• Maxmiize Performance 

• Maximize competition and innovation 

• Shift in risk 

• Achieve Savings 

The above objectives are achieved through performance requirement analysis, 
contractor performance assessment, contractor incentives, and market research, 

a. Performance Requirement Analysis 

This aspect of PBSA assists the team in identifying and defining desired 
outcomes and requirements. Overall, what needs to be accomplished is viewed from a 
top-level perspective. These outcomes are clearly defined as performance objectives and 
are included in a performance-based work statement and measurable performance 
standards are established. Figure 5 is a tree diagram that displays the flow of how desired 
performance outcomes are identified. 
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Figure 5. Tree Diagram for desired performance outcomes for Web portal development 

b. Market Research 

Market research is conducted wherein a Performance Based Services 
Acquisition (PBSA) team collects and analyzes information on commercial capabilities, 
processes, pricing, incentives, warranties, and delivery and other standard terms and 
conditions. This information is needed in order to determine the suitability of the 
marketplace for satisfying a need or requirement. The ultimate goal of market research is 
to help the acquisition team become informed consumers (Department of Defense 
Undersecretary for Acquisition Technology and Logistics, 2000) 

c. Contractor Incentives 

Incentives are a critical aspect of PBSA. Incentives, based on cost, 
schedule or quality of performance, are used to motivate superior contractor performance. 
Incentives can lead to improved competition among contractors, which can result in 
greater cost savings and value to the government. The types of contractor incentives are: 
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• Cost 

• Performance 

• Delivery 

• Mixed 

According to section 16.402 of the Federal Acquisition Regulation (FAR), 
Cost incentives, “take the form of a profit or fee adjustment formula and are intended to 
motivate the contractor to effectively manage costs.” This approach offers a potential 
cost-savings for the government. 

Performance Incentives are related to the product’s performance 
characteristics. Certain performance criteria are established, these criteria are assessed 
through evaluation and testing. This type of incentive can also be related to the 
contractor’s performance quality. An example of performance incentive is the speed of a 
ship. The Navy sets a threshold of 25 knots, which the contractor must achieve to meet 
the terms of the contract. In addition, the Navy sets an objective of 30 knots as the 
performance target for its new class of Patrol Craft. The Contractor earns the incentive 
fee by achieving performance up to the Navy’s objective value. 

Delivery incentives are most often used when the project schedule is 
primary priority for the contracting organization. For example, if troops require a new 
type of bulletproof vest within ten months, and the contract is awarded to deliver within 
nine, but includes an incentive fee for early delivery. 

Mixed incentives, refer to the combination of cost, performance, or 
delivery incentives. 

d. Contractor Performance Assessment 

Performance assessment is conducted through periodic performance 
evaluations of the contractor. This is to ensure that the government is receiving quality 
products and services in accordance with the defined desired outcomes listed in the 
performance work statement. Performance-based service acquisition has many benefits. 
They include: 

• Increased likelihood of meeting mission needs 

• Focus on intended results, not process 
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• Better value and enhanced performance 

• Less performance risk 

• Contractor flexibility in proposing solution 

• Better competition: not just contractors, but solutions 

• Shared incentives permit innovation and cost effectiveness (Integrated 

Acquisition Environment, 2010) 

The DoD recognized the benefits of the PBS A strategy when acquiring services. 
In 2000, the Undersecretary of Defense for Acquisition Technology and Logistics 
(AT&L), J.S. Gansler stated, “It is the policy of the Department of Defense (DoD) that, in 
order to maximize performance, innovation, and competition, often at lower cost, 
performance-based strategies for the acquisition of services are to be used wherever 
possible...to ensure that the Department continually reali z es these savings and 
performance gains, I establish, at a minimum, that 50 percent of service acquisitions, 
measured both in dollars and actions, are to be performance-based” (Gansler, 2000). 

D. DOD INFORMATION AND ACQUISITION GOALS AND 

REQUIREMENTS 

DoD has expressed goals for achieving information superiority without specifying 
how the IT systems it builds should accomplish that goal. DoD has a great need for 
incremental approaches, because it has significant investment in current systems and a 
limited budget for innovation (Hayes-Roth, 2003). In order to keep pace with growing IS 
requirements DoD’s vision “of a more agile and integrated organization whose systems 
are aligned with strategies, requires a shift from the existing approach of isolated "stove- 
piped" requirements development to an environment in which organizations embrace 
cross-community development,” must be paired with a more robust acquisition process 
that strategically plans its investment in IT infrastructure (Carey, 2009). 

The Department of Defense has indicated that the current acquisition process is 

inadequate for fielding mature, reliable IT systems rapidly enough to meet mission needs. 

Additionally, the spending associated with procuring these systems can often be well 

beyond what is necessary to meet mission requirements. Therefore, it is necessary to 

marry Strategic IT goals with a well-executed acquisition strategy to ensure that user- 

defined requirements are met in a reasonable timeframe, and within budgetary 

21 


constraints. In the realm of IT systems, wherein technology maturation occurs at an 
extremely rapid pace, current IT acquisitions must be developed with the capability of 
leveraging mature technologies to continue to meet mission requirements and avoid 
obsolescence. In order to facilitate these goals, acquisition reforms and legislation has 
been passed to provide guidance to acquisition decision makers. 

1. Clinger-Cohen Act of 1996 

The Clinger-Cohen Act (CCA) legislation outlines the Director of Office of 
Management and Budget (0MB) duties in ensuring the efficient usage of IT systems in 
accomplishing Federal responsibilities. It provides guidance for executive considerations 
for more efficient acquisition of information systems and services. It further indicates the 
designation of useful executive agents tasked with developing and instituting best 
practices in the acquisition of information technology (Office of Management and 
Budget, 1996). Notable requirements of the Act are: 

• The designation of a Chief Information Officer (CIO) 

• The requirement to balance technology standards and technology spending 

• The implementation of results-oriented management by establishing strategic 
goals 

• Ensuring a clear, simplified and understandable IS acquisition process 

• Establishment of a DoD-wide technical architecture to avoid fragmented 
information systems 

With regards to IS acquisition, the CCA calls for a clear and understandable 
process that specifically addresses the management of risk, incremental acquisitions, and 
the need to incorporate commercial IS in a timely manner (Office of Management and 
Budget , 1996). IS capability delivered incrementally presents an effective solution for 
managing evolving user requirements, technology maturation, and mission changes. The 
document also refers to modular contracting as a method to acquiring incremental 
technology that is highly cohesive yet loosely coupled. 

To meet the notable requirements outlined above, the CCA recommends 
performance-based and results-based management. The CCA clearly identifies activities 
that components should consider when making investments in Information Technology: 
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(B) Determine before making an investment in a new information system 

(i) Whether the function to be supported by the system should be 
performed by the private sector and, if so, whether any component of the 
executive agency performing that function should be converted from a 
governmental organization to a private sector organization; or 

(ii) Whether the function should be performed by the executive agency 
and, if so, whether the function should be performed by a private sector 
source under contract or by executive agency personnel (Office of 
Management and Budget, 1996) 

The CCA provides DoD components with a top-level perspective of government 
expectations for the effecive management and acquisition of IS assets. 

2. Modular Open Systems Approach (MOSA) 

A Modular Open Systems Approach (MOSA) is both a business and technical 
strategy for developing a new system or modernizing an existing one. DoDD 5000.1 
states that, “Acquisition programs shall be managed through the application of a systems 
engineering approach that optimizes total system performance and minimizes total 
ownership costs. A modular, open-systems approach shall be employed, where feasible.” 
MOSA supports achieving the following: 

• Reduced acquisition cycle time and overall life-cycle cost 

• Ability to insert cutting edge technology as it evolves 

• Commonality and reuse of components among systems 

• Increased ability to leverage commercial investment 
(Open Systems Joint Task Force, 2004) 

MOSA allows for evolutionary acquisition of systems. Systems are developed 
modularly to allow for greater ease of development, maintenance, modification and 
upgrade. This modularity mitigates impact on the system as system evolution and 
upgrade occurs over time. The primary enabler in MOSA are open standards. Open 
Standards, are selected based on their maturity, commerical acceptance, and allowance 
for future technology insertion. Standards for interface between current and future 
systems, specify the physical, functional, and operational relationships between various 
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elements namely, hardware and software. This is done to improve logistics support and 
permit interchangeability, interconnection, compatibility and/or communication (Open 
Systems Joint Task Force, 2004). 

The DoD requires greater collaboration among services; therefore systems that 
can function in the joint environment, and meet the needs for information sharing must be 
acquired. To meet this need, greater consideration must be given to IT investment 
strategies to ensure the DoD takes maximum advantage of the benefits available from 
acquiring and managing commercial software as a DoD enterprise asset (Money, 2010). 
DoD recognizes that greater buying power can be leveraged for the acquisition of IS and 
software if it is managed at a higher level. 

E. DON INFORMATION STRATEGY AND IT ACQUISITION GOALS 

Following the lead of the DoD, the Department of the Navy (DON) has called for 
a re-alignment in how its information systems (IS) integrate with one another in order to 
fulfill greater needs for collaboration among different mission areas, and more seamless 
integration of business processes. DON has taken a much more strategic enterprise¬ 
centric approach to IT integration and acquisition. The Department of the Navy has a 
requirement for development of an enterprise architecture that includes the management 
and resourcing of key enterprise services (USN, 2003). 

I. Enterprise Architecture (EA) 

Enterprise Architecture (EA) is a conceptual blueprint that defines the structure 
and operation of an organization. The purpose behind an Enterprise Architecture is to 
determine the most effective means for an organization to achieve its current and future 
objectives. EA is helpful for defining the core business processes within an organization. 
It provides a framework for aligning those processes in a way that accomplishes the 
organizations overall mission. When an organization leverages IS resources that fit 
within the overall architecture, the organizations ability to achieve its vision is greatly 
enhanced. 
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The EA can define the relationships between information systems (IS) that are 
grouped into mission areas within an organization. These groupings of systems provide 
mission capabilities that the organization leverages to meet its goals. The EA contains 
systems that can work autonomously of one another, yet the whole relies upon the 
systems interaction to meet core process requirements. It is important for any 
organization, especially those within the DoD to identify those systems that are critical to 
their operations as mission essential. A 2001 memorandum from Acting Secretary of the 
Navy Robert B. Eire Jr. defines Mission Essential Information Systems (MEIS) as, “A 
Mission Essential Information System is a system that meets the definition of 
"information system" in the Clinger-Cohen Act, that the acquiring Component Head or 
designee determines is basic and necessary for the accomplishment of the organizational 
mission. A Mission Essential Information Technology System has the same meaning as a 
Mission Essential Information System” (Eire, 2001). 

The DON has indicated that EA is the strategic approach to IS that it is instituting. 
In the DON Chief Information Officer’s (CIO) 2009 memorandum on Department of the 
Navy Enterprise Architecture Strategy, he states, “Eederally mandated EAs are a 
strategically-based means for DoD and DON to capitalize upon existing technological 
assets and make informed decisions about investments in new technology, in support of 
the war fighter. He further outlines the expected benefits of the DON EA. Expected 
strategic values from using the DON EA are to: 

• Ensure compatibility, flexibility, and interoperability among all DON 
networked elements. 

• Support the Capital Elanning and Investment Control (CEIC) process to ensure 
the Department's mission is achieved through consistent decision-making 
processes for IS investments. 

• Erovide consistent support to the critical decision-making processes of the 
DoD and DON (e.g.. Joint Capabilities Integration Development System 
(JCIDS), Defense Acquisition System (DAS), and Elanning Erogramming 
Budgeting and Execution (EEBE)). 

• Improve and promote broad use of common information sharing to ensure 
users can locate and access the right information at the right time. 
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• Leverage a Service Oriented Architecture (SOA) strategy, which organizes IS 
capabilities into interoperable, standards-based services that can be combined 
and reused to meet changing needs and threats. 

• Support alignment of activities, processes, systems and data to other DoD 
components and government agencies. 

Enterprise Architecture allows the Navy to facilitate information flows throughout 
its organization, even to disparate comers. It does this by integrating stove-piped systems 
with interconnected information systems to allow all assets to access necessary 
information resources for improved efficiency at mission accomplishment. EA is the 
DON strategy of choice, therefore all organizations within the department should strive to 
ensure their IS fit within that framework and organize themselves in a similar manner. 

2. Strategic Planning for IS investment 

Strategic planning for investing in IS and/or services should ideally consider an 
organization’s long-term goals, and current business process activities. An organization 
must continually assess current and future needs to identify which IS investment will 
meet those requirements. It is important that these systems can balance their total overall 
cost with delivery of desired effectiveness. Therefore, strategic plans should consider 
software, hardware, funding, and long-term vision in order to be effective. 

As for the DON, obtaining better control of IS investment is an acquisition goal. 
In 2009, the DON set forth an Information Technology Enterprise-Wide Investment 
policy that mandates greater leverage of DON buying power. This is power that 
components within DON can access. The policy is in place “to ensure that the DON 
speaks with one coordinated voice to suppliers when negotiating for most favorable terms 
and prices on products and services,” which is applicable to acquisition vehicles that are 
multi-claimant, meaning systems that can support and be used by more than a single 
claimancy (Carey, 2009). 

F. BENEFITS OF DEVELOPING A ROBUST IS ACQUISITION STRATEGY 

The Acquisition Strategy allows for a top-level view into the acquisition of an IT 
system. By pairing an AS with a detailed IS Strategy, an organization can better leverage 
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its buying power to acquire systems and services that will offer competitive advantages in 
prescribed core mission areas. The AS permits stakeholders to consider user needs and 
deliver a timely system that will meet those requirements within cost constraints. 
Additionally, an AS can be flexible to address evolving requirements due to mission 
changes. However, this flexibility must be closely managed to ensure that requirements 
changes do not result in cost overruns. 

G. SUMMARY 

Aligning an organization’s IS strategy with a well-defined and robust acquisition 
strategy will result in delivering a system that will meet mission requirements when 
needed, and within funding parameters. There are several acquisition strategies that can 
be used to achieve an organization’s IS goals. The most commonly used strategy within 
DoD is the Evolutionary Acquisition Strategy. However, Performance-Based Service 
Acquisition and Product Line Acquisition offer unique benefits for organizations such as 
the DoD that acquire many systems and services versus develop. The DoD has used the 
LAS and PBS A approaches to acquire IS. The PEA approach offers some benefits that 
should be considered, since the DON has identified Enterprise Architecture as its primary 
IS strategy. 
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III. BACKGROUND OF CNSF ORGANIZATION AND 
APPROACH TO IT ACQUISITION 

A. BACKGROUND OF COMMANDER, NAVAL SURFACE 

FORCES/COMMANDER, NAVAL SURFACE FORCES PACIFIC 

(CNSF/CNSP) 

I. History 

This section will present an overview of the origins and history of CNSF. Further 
exploration will be provided of the transformation of the command from its inception to 
its current role and responsibilities in the national defense structure. 

In 2001, the Chief of Naval Operations designated Commander In Chief, Atlantic 
Fleet, as Commander, Fleet Forces Command (CFFC). CFFC responsibilities would 
include coordination, establishment, and implementation of integrated requirements and 
policies for manning, training, and equipping both Atlantic and Pacific Fleet assets during 
their inter-deployment training cycle. Type Commanders (TYCOMs) within each 
warfare area would support CFFC in executing these tasks. Commanders, Naval Surface 
Forces Pacific, Naval Air Force Pacific, and Naval Submarine Force Atlantic assumed 
duties as TYCOMs. These Fleet TYCOMs became knows as Commander, Naval Surface 
Forces (CNSF), Commander, Naval Air Forces (CNAF), and Commander, Naval 
Submarine Forces (COMNAVSUBFOR) (Commander Naval Surface Forces U.S. Pacific 
Fleet, 2009). 

In 2004, CFFC established echelon three TYCOM lead organizations as three-star 
lead and two-star deputy TYCOMs on opposite coasts. Commander, Naval Surface 
Forces Pacific (CNSP) was established as the lead three-star and dually responsible for 
Pacific Surface forces as well as all Navy surface assets. Commander, Naval Surface 
Forces Atlantic (CNSL) became the two-star TYCOM. The policy was initially 
established as CNSF providing policy and guidance and CNSL providing current 
readiness oversight for all Surface Forces ships (Commander, Naval Surface Forces U.S. 
Pacific Fleet, 2009). 
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In 2005, the Surface Warfare Enterprise was established. The SWE mission 
included development of technically superior forces through well-maintained combatant 
crafts and highly trained personnel, prepared to quickly deploy in support of Combatant 
Commander tasking requirements (Commander Naval Surface Eorces U.S. Pacific Elect, 
2009). 

In 2006, The Chief of Naval Operations disestablished CEEC and Commander, 
Atlantic Elect and renamed it to Commander, U.S. Elect Eorces Command (CUSEEC). 
Also in 2006 the SWE stood up and reinforced that CNSE was the Current Readiness 
Officer and established Class Squadrons as direct reports under CNSE. 

In 2007, CUSEE established the Elect Readiness Plan (ERP) to replace the inter¬ 
deployment training cycle (Commander Naval Surface Eorces U.S. Pacific Elect, 2009). 

Commander Naval Surfaces Eorces (CNSE) resulted from a 2001 CNO driven 
transition in the command structure. 

2. Mission 

The mission of CNSE is to man, train, and equip ships and other surface assets 
within the fleet with superior, technologically equipped surface platforms, and highly 
effective, well-trained sailors. These tasks are accomplished through managed readiness 
and training. 


3. Responsibilities 

The re-alignment of the CNSE structure and organization resulted in greater 
responsibility for the CNSE organization. Dually assigned as CNSE/CNSP, CNSE 
functions as the TYCOM for the surface ships in the Pacific to Commander, U.S. Pacific 
Elect (COMPACEET) for executing U.S. Code Title 10 responsibilities for manning, 
training, and equipping those ships. In addition, it answers to CUSEEC for any surface 
warfare request for forces. Einally, as the Pacific lead, it tracks Pacific ship readiness via 
West Coast Immediate Superior-In-Charge (ISICs) and Class Squadrons (CEASSRONs) 
(Commander Naval Surface Eorces U.S. Pacific Elect, 2009). CNSE is Director of the 
Surface Board that governs the Surface Warfare Enterprise (SWE). CNSE/CNSP Reports 
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directly to CUSFF for Surface Force Warships Ready for Tasking to meet Combatant 
Commanders (COCOM) Request for Forces (RFF) requirements, and to CPF as Surface 
TYCOM for Pacific Fleet warships, as well as readiness trends and process improvement 
efforts to the Fleet Readiness Enterprise (FRE) as the SWE Commander, and sets 
ultimate SWE strategic direction (Commander Naval Surface Eorces U.S. Pacific Elect, 
2009). 

B. BACKGROUND OF COMMANDER, NAVAL SURFACE FORCES 

ATLANTIC (CNSL) 

1. History and Mission 

Currently, CNSL functions as Deputy CNSE. The mission of Commander, Naval 
Surface Eorce, U.S. Atlantic Elect, is to provide combat ready ships and stations to the 
Elect, and to ensure that those ships and stations are supplied the leadership, manpower, 
equipment, maintenance, training, and material needed to quickly achieve decisive 
victory at and from the sea. The Naval Surface Eorce Commander prescribes readiness 
and training requirements for assigned Eorces, and ensures that deploying units meet 
prescribed readiness standards. The Surface Eorce Command includes all of the ships in 
the Atlantic and Mediterranean Elects, with the exception of aircraft carriers, submarines, 
submarine support ships, and Military Sealift Command vessels (Commander, Naval 
Surface Eorces, 2005). 

2. Responsibilities 

As Deputy CNSE, CNSL Reports directly to CNSE as his Eorce Deputy and also 
directly to CUSEE as Surface TYCOM for warfighting readiness of Atlantic Elect ships. 
Additionally, Acts as Chief Readiness Officer (CRO), monitoring battle readiness of all 
surface ships, tracking metrics and analyzing trends via CLASSRONs, providing results 
and collaborating deep dive requirements with SWE Cross Eunctional Teams (CETs). 
Monitors and tracks battle readiness via Atlantic (LANT) Elect ISICs and CLASSRONs 
for basic phase certifications and across Elect Readiness Programs (ERP) for LANT Elect 
ships (Commander, Naval Surface Eorces U.S. Pacific Elect, 2009). CNSL shares 
administrative responsibility for training and operational readiness of the Atlantic forces. 
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C. BACKGROUND OF COMMANDER, FLEET FORCES COMMAND 
(CFFC) 

1. Mission 

Commander, Atlantic Fleet (COMLANTFLT) was established in 1975 as the 
TYCOM for Operational training and tasking for the U.S. Atlantic Fleet forces. The 
Command was re-designated United States Fleet Forces Command (USFFC) in 2001. 

United States Fleet Forces Command supports both the Chief of Naval Operations 
and Combatant Commanders worldwide by providing responsive, relevant, sustainable 
Naval forces ready-for-tasking. The command provides operational and planning support 
to Combatant Commanders and integrated warfighter capability requirements to the 
CNO. Additionally, U.S. Fleet Forces Command serves as the CNO's designated 
Executive Agent for Anti-Terrorism/Force Protection (ATFP), Individual Augmentees 
(lA), and Sea Basing. In the USFF Guidance, the Commander discusses the following 
CFFC authorities: 

• Authority to organize, man, train, maintain and equip assigned Navy forces; 

• Budget Submitting Office (BSO) authority and responsibility for assigned 
forces, military and civilian personnel, infrastructure, and budget; and 

• Authority to generate and communicate Navy global force management 
solutions concerning general purpose forces, ad hoc forces and individual 
augments to Commander, U.S. Joint Forces Command. (Harvey, 2009) 

In collaboration with U.S. Pacific Fleet, U.S. Fleet Forces Command organizes, 
mans, trains, maintains, and equips Navy forces, develops and submits budgets, and 
executes readiness and personnel accounts to develop both required and sustainable 
levels of Fleet readiness. Additionally, the command serves as the unified voice for Fleet 
training requirements and policies to generate combat-ready Navy forces per the Fleet 
Response Plan (FRP) using the Fleet Training Continuum (FTC) (Navy PAO, 2010). To 
meet the requirements of Fleet training and readiness CFFC implemented the Fleet 
Readiness Enterprise (ERE). 
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2. Fleet Readiness Enterprise (FRE) 

To meet readiness requirements mandated in the Navy’s FRP, CFFC implemented 
the FRE. The FRE assists in meeting requirements for readiness against increasing 
varieties of threat, while balancing limited resources. The ERE was formed as a result of 
the Navy’s adoption of an Enterprise approach to the “business” of military operations. 
The ERE focuses on asset integration, and effective usage of those assets despite stagnant 
resources, to provide a well equipped force ready for any tasking. 

The ERE receives input from the various warfare enterprises. The structure of the 
subordinate warfare enterprises make up the readiness enterprise which represents 
readiness for the Navy Enterprise in the national defense structure. A dashboard system 
known as the Defense Readiness Reporting System (DRRS) provides top-level 
commanders with an updated view of force readiness for tasking. The Navy’s dashboard 
is known as DRRS-N. Eigure 6 depicts the organization of the ERE. 
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Figure 6. Supporting Structure of Fleet Readiness Enterprise (From Navy Enterprise, 

2010 ) 

A board, headed by the TYCOM, governs each Warfare Enterprise. The 
TYCOMs are: 

• Commander, Naval Surface Eorces (CNSE) 

• Commander, Naval Air Eorces (CNAE) 

• Naval Network Warfare Command (NETWARCOM) 

• Commander, Naval Submarine Eorces (COMNAVSUBEOR) 

These TYCOMS such as CNSE are frequently required to operate within a gray 
area between Title 10 TYCOM accountability issues and overlapping Enterprise 
responsibilities (Commander Naval Surface Eorces U.S. Pacific Elect, 2009). Eigure 7 
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displays a scaled view of TYCOM organization and interaction in the Navy. Figure 8 
depicts daily Title 10 exeeution of the CNSF organization. 



Figure 7. Type Commander Organization (From Navy Enterprise, 2010) 
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Figure 8. TYCOM Title 10 daily exeeution 

The Warfare Enterprises are: 

• Surface Warfare Enterprise (SWE) 

• Naval Aviation Enterprise (NAE) 

• Undersea Enterprise (USE) 

• Naval Netwar/EORCEnet Enterprise (NNEE) 

• Naval Expeditionary Combatant Enterprise (NECE) 

Each of these Warfare Enterprises supports the ERE in delivering combat 
capability to Navy Components and Combatant Commanders, via the training processes 
of Commander Second Elect and Commander Third Elect (Navy Enterprise, 2010). 
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D. SURFACE WARFARE ENTERPRISE (SWE) 

The SWE is made up of all Surface Warfare partners, from the people at Naval 
Sea Systems Command who research, develop and acquire our equipment; to the officers 
on the Navy staff responsible for funding; and commands responsible for logistics, 
personnel policy, maintenance support, and more. The SWE brings the Surface Warfare 
Community tools to improve processes, execute streamlined business practices, and gain 
efficiencies throughout our Navy to produce improved war fighting readiness (Surface 
Warfare Enterprise, 2010). The SWE is comprised of a Surface Board and headed by 
CNSE. Its mission is to ensure the fleet is prepared for emergent tasking, and available 
for combatant commanders to employ when necessary. The SWE is organized into 
Cross-Eunctional Teams (CETs) that report to the Surface Board via the SWE Deputy on 
supporting elements vital to the operational readiness of Surface Eorces. The CETs are: 

• Euture Capabilities Team (ECT) 

• Euture Readiness Team (ERT) 

• Personnel Readiness Team (PRT) 

• Strategic Einancial Management Team (SEMT) 

• Current Readiness Team (CRT) 

Eigure 9 from a 2008 CNSE SWE Brief displays the SWE structure and 
interaction between the various elements within the enterprise. 
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Figure 9. SWE Structure as of 2010 (From Pena, 2010) 


1. SWE Systems and Capabilities 

The SWE mission of ensuring the operational readiness of the Navy Surface 
Force is achieved through leveraging IS. The IS provide leadership with a dashboard 
view resulting in improved decision-making capability. The dashboard views are a result 
of applications that have been contracted into services within the past ten years. The 
primary application services used by CNSF are: 

• Training and Operational Readiness Information Services (TORIS) 

• Continuous Monitoring Program (CMP) 

• Surfaces Forces Web (SURFOR Web/CNSF Web) 

a. TORIS 

The TORIS system of applications focuses on all aspects of training to 

support operational readiness for the surface force. Using a Web-based portal, TORIS 
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provides high-level visibility into the training status of the Navy’s surface force down to 
the unit level perspective. It is designed to assist commanders with the information 
management challenge they are faced with daily. TORIS is classified as an authoritative 
data source that “collects, stores, and displays proficiency and certification data from the 
surface fleet" (Commander, Naval Surface Forces, 2009). 

TORIS was designed and implemented following the 9/11 terrorist attacks, 
after the need for an instant view of surface fleet readiness was identified. The need 
stemmed from the greater requirement for identifying surface units that were ready for 
Combatant Commander (COCOM) tasking. As a result, TORIS provided Unit 
Commanders with visibility into their unit readiness as compared to Afloat Training 
Group (ATG) requirements and metrics (Roberts, 2010). 

TORIS is a vital component to the CNSF mission. It has automated key 
business processes to meet command mission requirements. Since its implementation the 
Service has enabled ATG to meet mandated reduced manning requirements and allowed 
ATG Ship Trainers and Evaluators to greater focus to the operational readiness of the 
Navy’s surface assets, therefore greatly contributing to CNSF/SWE command goals. 
TORIS provides input to the “T” Pillar for training to the DRRS-N system to report force 
readiness to the national structure via the Defense Readiness Reporting System (DRRS), 
this is displayed in Eigure 10, which was designed by the TORIS Project Team at ATG. 
The service integrates CNSE training requirements promulgated via the Surface Eorce 
Training Manual (SETM). TORIS uses three processes gathering data used in 
populating the “T” pillar of DRRS-N: 

• The Onboard training record, which gives unit commanders a view into 
their unit readiness. 

• The periodic inspection and certification process for surface units. The 
goal is to decrease the time units spend under inspections and increase 
the time they are operational and ready for tasking. 

• Automated DRRS-N population 
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Figure 10. Relationship between TORIS and DRRS-N “T” Pillar (From TORIS ppt 

2008) 


While TORIS and its data is used primarily by CNSF subordinates for 
various Business Mission Areas (BMAs), there are other organizations outside of the 
FRF that are benefitting from the data resources that the service provides. For example, 
according to TORIS managers the program frequently receives data calls from OPNAV 
N7 in support of Navy Staff research efforts. CNSF also has a Memorandum of 
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Agreement (MOA) with the United States Coast Guard (USCG) to support the surface 
training and operational readiness of their forces. 

In 2009, the TORIS contract cost approximately $2.3M, which has been 
consistent throughout the life of the program. The TORIS funding covers system 
maintenance, manning requirements, and further system development. Ultimately the 
system effectively meets requirement set forth by both CFFC in the FRP and CNSF 
training requirements guided by the SFTM. 

b. CNSF Web/SURFOR Web 

SURFOR Web provides a Web-based collaboration and knowledge 
management capability for CNSF surface assets. A 1999 U.S. Pacific Fleet staff 
knowledge management study revealed the need for a Web-based collaboration resource 
that would serve as the central location for organizational knowledge vital for mission 
support. 2004 marked the launch of the consolidated Commander Naval Surface Forces 
Pacific (CNSP) and Commander Naval Service Forces Atlantic (CNSL) Web portal. By 
2007 CNSF required all public and private Websites of its subordinate commands to 
migrate to the SURFOR Web portal. 

SURFOR Web is currently the central location for policy documents, 
directives, and most COMNAVSURFOR readiness tracking metrics (KPFs, FOMS, 
trends, costs, etc.). CNSF Web is the primary tool used to promulgate this information to 
ships and shore-based commands (Griffith). SURFOR Web’s primary role is alignment 
of the force. It meets the CNO’s FORCEnet, “Alignment” and Sea Enterprise goals as 
well as the business requirements of the Surface Eorces Enterprise (SWE). SUREOR 
Web’s evolution: 

2004: The SUREORWEB portal NIPRNET/SIPRNET came on-line in 
October 2004 with the consolidated CNSP/CNSE collaboration site. 

Within a few months an Enterprise strategy was formulated and SUREOR 
commands were given the option of consolidating with the SUREOR 
Web. 

2005: In response to the Cyber Condition Zebra initiative (DTG I0I926Z 
MAY 2005), CNSE consolidated many of the claimancy legacy systems 
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into the SURFORWEB environment including the Continuous Monitoring 
Program (CMP) and ATGLANT Toolkit and Level of Knowledge 
Program (LOK) 

2006: As part of the Cyber Condition Zebra initiative the SURFORWEB 
NIPRNET portal was migrated from the SPEAR network to the NMCI 
Enterprise Network in August 2006. A formal plan was documented in 
JUN 06 to consolidate the SURFOR Web with USFF’s Fleet Forces 
Online (FFO) Portal. 

2007: In response to NAVADMIN 145-07, CONSOLIDATION OF 
NAVY WEBSITES REDUCTION OF IM/IT FOOTPRINT, CNSF 
mandated that all Public and Private Websites migrate to the 
SURFORWEB environment. This consolidation was completed in 
November 2007. 

2008: SURFORWEB NIPRNET received full accreditation (ATO) from 
NETWARCOM July 2008. 

2008/09: SURFORWEB SIPR is awaiting Interim Authority to Operate 
(lATO) via Certification & Accreditation (C&A) submitted 15 Dec 08 

2009: NETWARCOM DEC 08 Public Website Migration Supplemental 
Guidance 

(Commander, Naval Surface Forces, 2009) 

CNSF Web provides SWE staff and command units with a secure, central 
location for integrated Web-based collaboration. It is a resource for collaboration of vital 
CNSF business processes. To include: 

• Publicly Accessible Websites 

• Family Websites 

• Private Business Collaboration sites 

• Departure From Specifications Database (DFS) 

• C5 Readiness Assessment (C5RA) 

• Authorized Equipment Listing Program (ALL) 

• COMETH 

• Hot Wash 

• 2MCAL 

• War Fighting Improvement Program (WFIP) 

• Continuous Monitoring Program (CMP) 

• ATGLANT Toolkit and Level of Knowledge Program (LOK) 

(Commander, Naval Surface Forces, 2009) 
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Current funding of CNSF Web is approximately $2.2M. The program is 
meeting the command’s need for collaboration on both classified and unclassified 
enclaves. As a knowledge management tool, SURFOR Web is a key resource supporting 
the execution of some of its core business processes. 

E. BACKGROUND OF COMMANDER NAVAL AIR FORCES (CNAF) AND 
THE NAVAL AVIATION ENTERPRISE (NAE) 

1. History 

Much like CNSF the Type Commander (TYCOM) for the surface forces of the 
fleet. Commander Naval Air Forces (CNAF) is tasked as TYCOM to represent the 
specific needs of the Navy’s Aviation Force. The Command’s beginning was nearly 
identical to that of CNSF in that there are command’s on both coast; Commander Naval 
Air Forces Atlantic (CNAL) and Commander Naval Air Forces Pacific (CNAP). 
However, the Chief of Naval Operations placed Type Commanders (TYCOMs) 
in a "Lead-Follow" arrangement in October 2001. Under this arrangement, 
COMNAVAIRPAC became TYCOM for Air and assumed the additional title 
of Commander, Naval Air Forces (COMNAVAIRFOR/CNAF) (PAO, 2010). 

2. Mission 

According to the CNAF command Website, the CNAF mission is “to man, 
train, equip and maintain a Naval Air Force that is immediately 
employable, forward deployed, and engaged. We support the Fleet and 
Unified Commanders by delivering the right force with the right readiness 
at the right time with a reduced cost ... today and in the future. (PAO, 
Commander, Naval Air Forces, 2010) 

3. Naval Aviation Enterprise (NAE) 

The NAE was formed as a partnership between multiple organizations within 
Naval Aviation to resolve interdependent issues that affect multiple commands 
(Williams, 2007). The NAE structure is very similar to the organizational formation of 
the SWE. An Aviation Board made up of Naval Aviation resource providers and 
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supporters make eritieal deeisions as to the direetion and future of the Navy’s Aviation 
eommunity. Figure 11 displays the strueture and the interaetion among the various 
elements within the NAE. 
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Figure 11. Naval Aviation Enterprise Strueture (From Etter, 2006) 


The NAE “supports the readiness requirements of Naval Aviation by 
enhaneing eommunieation, fostering organizational alignment, 
encouraging inter-service integration, stimulating a culture of productivity, 
and facilitating change when change is needed to advance and improve. 

The NAE’s single fleet-driven metric is: Naval Aviation forces efficiently 
delivered for tasking. (Naval Aviation Enterprise, 2010). 

4. Naval Aviation Enterprise (NAE) Systems 

a. SHARP 

SHARP has tracked aviation training and readiness since 2000. Originally 
designed for tracking helicopter readiness, now it has widespread use, and its metrics 
have been used to defend flight hours. 

The programs effectiveness can be attributed to a couple of factors. First, 
the performance factors are based on events that are required to achieve combat 
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readiness. Secondly, consolidated reports are captured monthly and cataloged in an 
aviation data warehouse. Finally, multiple are taken into considerations to determine 
overall readiness. 

The baseline has been evolving since its inception, currently strides are 
being made to lock down the current code. The ultimate goal is to develop a solid 
baseline and build modules as they are needed. Currently, enhancements are made to 
meet new user requirements. So far these changes have not been a financial burden 
because they are funded by the requesting agency. SHARP provides such features such 
as; log planning, logging of flight hours, and tracking dollars spent for programs. 

b. Carrier Sierra Hotel Readiness Reporting Program (CV SHARP) 

CV SHARP tracks carrier training and readiness down to the sailor level. 
It was modeled after the SHARP program, which had already proven the worthiness of 
tracking training. It was established in 2005 and its first true baseline of services was 
established in 2009. 

It was initially used internal to carriers as a training tool. Since its 
inception, it has evolved from version 1.0 to 2.3 and is being installed on 9 carriers. This 
upgrade will allow those carriers the capability of reporting to DRRS-N, which is 
required by USFF. The funding for its evolution has come out of O&M funds from 
aviation flight hour money. Currently, this program’s budget has been successfully 
managed. However due to anticipate future funding cuts, the Program Manager (PM) 
predicts a future of maintenance with very little software development. 

The CV SHARP program provides NAE Board and resource supporters 
insight into the training and readiness of the Aviation Enterprise. As a result they are 
able to make decisions concerning resource allocation, and identify shortfalls. As a tool 
it is used track and report training accomplishments and gauge readiness for tasking 
(RET) of its Aircraft Carrier CV(N) platforms. 

In support of the Elect Readiness Plan (ERP), CV SHARP provides 
readiness inputs to the “T” Pillar in the Defense Readiness Reporting System-Navy 
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(DRRS-N) for status of readiness for aviation forces. It provides CV(N) Commanding 
Officers (CO) a view into the training of the sailors within his command. The CO can 
use this information to predict the impact of shortfalls on his crew’s operational 
capability (LeFon, 2009). 

CV SHARP was acquired using an Evolutionary Acquisition Strategy 
(EAS), and is currently contracted for a 10 year period resulting in a $4M/year contract. 

F. SUMMARY 

This chapter discussed the backgrounds and histories of United States Elect 
Eorces Command (USEEC) and subordinate Type Commanders (TYCOMs) Commander, 
Naval Surface Eorces (CNSE) and Commander, Naval Air Eorces (CNAE). The Surface 
Warfare Enterprise (SWE) structure, mission and vital information applications were 
discussed. The TORIS and CNSE Web programs were reviewed, as well as their 
contribution to the overall achievement of SWE goals. History and mission of CNAE 
and the Naval Aviation Enterprise (NAE) were outlined. The Carrier Sierra Hotel 
Aviation Readiness Program (CV SHARP) purpose and background was discussed. The 
CV SHARP program is a vital component to achieving the mandated Elect Readiness 
Plan (ERP). 
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IV. ANALYSIS AND FINDINGS 


A. INTRODUCTION 

This chapter will discuss both qualitative and quantitative findings discovered 
during this study. Both analyses will explore acquisition concepts and their applicability 
to how an organization functions in regard to the Acquisition Strategy Characteristics 
(ASC). The section closes with proposed solutions that connect the critical points 
associated with the ASCs. 

In the quantitative section of the chapter, ordinal data will be presented that will 
identify statistical responses given by various decision makers via comparison surveys of 
vital Information Systems (IS) within the Surface Warfare Enterprise (SWE) and Naval 
Aviation Enterprise (NAE). The surveys are based upon the ASCs introduced in chapter 
two. 

The section on qualitative analysis will compare the SWE and NAE goals and 
approaches toward acquiring similar ISs that are used in executing core business 
processes. It also offers a more in-depth analysis of the organizational goals and how 
those goals best fit with the Acquisition Strategies (AS) discussed in chapter two. 
Analysis of a GAO case study of companies in the private sector transformation in IS 
acquisition spending will provide a background for further recommendation for CNSE. 

B. QUANTITATIVE ANALYSIS OF CNSF AND CNAF INFORMATION 

SYSTEM (IS) PROGRAM ACQUISITION 

I. Introduction 

In the quantitative section of the chapter, ordinal data will be presented that will 
identify statistical responses given by various decision makers via comparison surveys of 
vital IS within the SWE and NAE. The surveys are based upon the ASCs introduced in 
chapter two. The final analysis and the proposed solutions connect the critical points of 
the survey and directly relate to the ASCs. 
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2. Survey Analysis 

The target programs for this analysis were the CNSF Web, TORIS, SHARP, and 
CV SHARP. This section is organized by the way the data is interpreted. First, the data is 
reviewed in terms of its mean and arranged by its frequency distribution. This provides a 
graphical overview of the survey results. The next section further defines the strength of 
the mean by considering each population’s standard deviation. Finally, due to the small 
population size, nonparametric techniques provide information about the populations and 
the measuring variables which are the ASCs. 

Table 1 shows how they responded to the questions. The ratings are based upon 
the following scale: 

• Very significant or very important (Rating: 1) 

• Significant or Important (Rating: 2) 

• Moderately Significant or Moderately Important (Rating: 3) 

• Little Significance or Little Importance (Rating: 4) 

• Not Significant or Not Important (Rating: 5) 

The survey measured the participants’ perceptions of how their program was 
affected by changes in policy, personnel, funding, and requirements. The questions in the 
survey are based upon the ASC’s presented in Table 1. In addition to the questions listed 
below, the survey also included questions about their programs integration, 
implementation, procurement, and certification. These questions captured time 
requirements and difficulty ratings and are located in Table 2. The data supporting these 
tables are located in Appendix A. 
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CNSF Web 

TORIS 

SHARP 

?/SHARP 

Realism 















What significance has the 
implementation of this program 
had on the Command’s overall 
mission? 

2.00 

1.00 

1.00 

1.00 

2.00 

4.00 

1.00 

1.00 

1.00 

1.00 

1.00 



Mean 

1.50 

1.57 

1.00 

1.00 



How significant has Command 
support been to the overall 
success of this system? 

3.00 

2.00 

1.00 

1.00 

1.00 

1.00 

1.00 

2.00 

1.00 

2.00 

1.00 



Mean 

2.50 

1.14 

2.00 

1.00 



How significant did force-fit 
measures, that can't be 
compromised, affect the 
development or operation of this 

program? 

4.00 

5.00 

4.00 

4.00 

1.00 

3.00 

2.00 

3.00 

1.00 

3.00 

2.00 



Mean 

4.50 

2.57 

3.00 

2.00 



How important have the role of 
priority levels, when compared to 
similar programs, affected 
decisions made for your 
particular program? 

4.00 

4.00 

2.00 

3.00 

2.00 

1.00 

3.00 

2.00 

1.00 

3.00 

2.00 



Mean 

4.00 

2.00 

3.00 

2.00 



To what extent were realistic 
expectations important with 
regard to this program's 
scheduling and resource 
requirements? 

3.00 

4.00 

2.00 

2.00 

1.00 

2.00 

1.00 

2.00 

1.00 

2.00 

1.00 



Mean 

3.50 

1.57 

2.00 

1.00 


Stability 















How significant will changes by 
the user or in capability cause a 
major disruption of the technical 
progress needed for the system? 

4.00 

5.00 

4.00 

4.00 

3.00 

2.00 

2.00 

3.00 

1.00 

4.00 

4.00 



Mean 

4.50 

2.71 

4.00 

4.00 



How significant will the changes 
in policy or leadership affect the 
direction of this project? 

3.00 

4.00 

3.00 

2.00 

1.00 

1.00 

3.00 

1.00 

1.00 

2.00 

2.00 



Mean 

3.50 

1.71 

2.00 

2.00 



How significant is the impact of 
having higher level supporters 
speaking on behalf of your 
program? 

1.00 

2.00 

1.00 

1.00 

1.00 

1.00 

2.00 

1.00 

1.00 

1.00 

2.00 



Mean 

1.50 

1.14 

1.00 

2.00 



How significant could the use of 
an IPT improve the success of 
this program? 

2.00 

4.00 

3.00 

3.00 

4.00 

3.00 

3.00 

2.00 

1.00 

3.00 

5.00 



Mean 

3.00 

2.71 

3.00 

5.00 



Table 1. Survey Feedback (1 of 3) 
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How would changes in industry, 
loss of a major contractor, or 
failure to modernize impact this 
program? 

1.00 

2.00 

1.00 

1.00 

2.00 

1.00 

1.00 

1.00 

1.00 

1.00 

2.00 



Mean 

1.50 

1.14 

1.00 

2.00 



How would changes in personnel 
that cause a lack of 
accountability or a loss of audit 
trail affect this program? 

2.00 

2.00 

1.00 

2.00 

2.00 

1.00 

2.00 

1.00 

1.00 

3.00 

1.00 



Mean 

2.00 

1.43 

3.00 

1.00 



How would changes in personnel 
that cause a lack of 
accountability or a loss of audit 
trail affect this program? 

2.00 

3.00 

5.00 

2.00 

3.00 

1.00 

2.00 

1.00 

1.00 

2.00 

1.00 



Mean 

2.50 

2.14 

2.00 

1.00 


Resource Balance 














How significant have the 
management of budget for the 
personnel, facilities, and time 
affected the successful 
development or operation of this 
project? 

2.00 

2.00 

2.00 

2.00 

1.00 

1.00 

1.00 

2.00 

1.00 

2.00 

1.00 



Mean 

2.00 

1.43 

2.00 

1.00 



How important was budgeting 
from inception to the overall 
execution of this program? 

2.00 

2.00 

1.00 

2.00 

1.00 

1.00 

3.00 

2.00 

1.00 

2.00 

2.00 



Mean 

2. 

OO 

1.57 

2.00 

2.00 



How important would being part 
of a continuous funding vehicle 
affect the financial stability of 
this project? 

2.00 

2.00 

1.00 

2.00 

1.00 

1.00 

3.00 

1.00 

1.00 

2.00 

2.00 



Mean 

2. 

00 

1.43 

2.00 

2.00 


Flexibility 















How significant would it be if the 
user/customer was readily 
available to answer questions? 

2.00 

3.00 

1.00 

2.00 

3.00 

1.00 

1.00 

1.00 

1.00 

3.00 

3.00 



Mean 

2.50 

1.43 

3.00 

3.00 



How significant would it be if the 
contract for the program had 
built in flexibility allowing 
changes to happen without 
consultation with the customer? 

2.00 

4.00 

4.00 

3.00 

4.00 

1.00 

2.00 

2.00 

1.00 

2.00 

2.00 



Mean 

3.00 

2.43 

2.00 

2.00 



How important has risk 
management planning been to 
maintaining the program 
schedule? 

4.00 

3.00 

3.00 

2.00 

2.00 

2.00 

3.00 

1.00 

1.00 

2.00 

1.00 



Mean 

3.50 

2.00 

2.00 

1.00 



How much of an impact to the 
design was attributed to ensuring 
that the program could support a 
balance among performance, 
productivity, and logistics? 

4.00 

2.00 

3.00 

2.00 

3.00 

1.00 

2.00 

2.00 

1.00 

3.00 

2.00 



Mean 

3.00 

2.00 

3.00 

2.00 


ManaKd Risk 1 I 

1 1 1 1 1 1 





Table 2. Survey Feedback (2 of 3) 
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How significant have 
uncontrollable requirement shifts 
affected this Broaram? 

3.00 

3.00 

5.00 

4.00 

2.00 

1.00 

2.00 

2.00 

1.00 

4.00 

1.00 



Mean 

3.00 

2.43 

4.00 

1.00 



How significant have the role of 
natural disasters affected the 
development or operation of this 
Broaram? 

4.00 

5.00 

5.00 

5.00 

5.00 

5.00 

4.00 

4.00 

1.00 

4.00 

5.00 



Mean 

4.50 

4.14 

4.00 

5.00 



What significance has funding 
changes had on the direction of 
this proBram? 

2.00 

4.00 

1.00 

3.00 

2.00 

1.00 

4.00 

2.00 

3.00 

3.00 

3.00 



Mean 

3.00 

2.29 

3.00 

3.00 



How significant have governmea^ 
standards affected the 
development or operation of this 
Broaram? 

2.00 

4.00 

1.00 

3.00 

1.00 

1.00 

3.00 

2.00 

3.00 

1.00 

2.00 



Mean 

3.00 

2.00 

1.00 

2.00 



How important have changes 
with the contractor affected the 
development or operation of this 
program? 

4.00 

1.00 

5.00 

2.00 

2.00 

1.00 

3.00 

1.00 

1.00 

2.00 

1.00 



Mean 

2.50 

2.14 

2.00 

1.00 



How important are the impacts 
of external directives that affect 
the development or operation of 
the prceram? 

2.00 

2.00 

1.00 

1.00 

1.00 

1.00 

2.00 

1.00 

1.00 

1.00 

2.00 



Mean 

2.00 

1.14 

1.00 

2.00 



How important has the use of 
mature technology been to the 
development or operation of the 
Broaram? 

2.00 

2.00 

1.00 

2.00 

1.00 

1.00 

2.00 

2.00 

1.00 

2.00 

1.00 



Mean 

2.00 

1.43 

2.00 

1.00 



How important has achieving 
reliability, availability, and 
maintainability impact the 
development or operation of this 
prosram? 

3.00 

1.00 

1.00 

1.00 

2.00 

1.00 

2.00 

2.00 

1.00 

2.00 

1.00 



Mean 

2.00 

1.43 

2.00 

1.00 



Table 3. Survey Feedback (3 of 3) 
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CNSF Web 

1 TORIS 

SHARP 

rVSHARP 

Continuous Variables 













Time to Implement 

2.00 

2.00 

2.00 

2.00 

2.00 

2.00 

2.00 

2.00 

2.00 

2.00 

5.00 


Mean 

2.00 

2.00 

2.00 

5.00 


Time to Integrate 

1.00 

1.00 

1.00 

1.00 

5.00 

1.00 

1.00 

1.00 

1.00 

1.00 

2.00 


Mean 

1.00 

1.57 

1.00 

2.00 


Time to achieve 

Certification 

1.00 

1.00 

1.00 

2.00 

1.00 

1.00 

2.00 

1.00 

2.00 

1.00 

2.00 


Mean 

1.00 

1.43 

1.00 

2.00 

Likert Scales 













Level of difficulty 
in Integration 

5.00 

6.00 

5.00 

4.00 

5.00 

5.00 

6.00 

5.00 

4.00 

5.00 

7.00 


Mean 

5.50 

4.86 

5.00 

7.00 


Level of difficulty 
in Procurement 

4.00 

5.00 

5.00 

5.00 

6.00 

5.00 

6.00 

5.00 

4.00 

3.00 

7.00 


Mean 

4.50 

5.14 

3.00 

7.00 


Level of difficulty 
in Certification 

6.00 

5.00 

3.00 

6.00 

4.00 

5.00 

6.00 

4.00 

5.00 

6.00 

7.00 


Mean 

5.50 

4.71 

6.00 

7.00 



1 

1 1 1 1 1 1 ~ 




Table 4. Survey Feedbaek: Time and Diffieulty Rating 


a. Population Description 

The mean ratings from the survey are measured in terms of frequency. The 
frequency distribution provides insight into the overall decision-making climate for each 
program (Schutte, 1977). This information can also help describe each population. The 
supporting data for the remaining quantitative analysis is located in Appendix B. 

(1) Acquisition Strategy Characteristics Survey Results. In 
figure 12, the chart below shows the frequency of occurrence for rating each question on 
a scale of “1” to “5”. The bars in the chart are based upon the mean values of their 
replies and the standard error indicating the difference between the sample size and 
population (Rumsey, 2003). The intersecting bracket on each bar indicates the amount of 

standard error. This chart indicates the following: 
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• The Mean rating from those surveyed for TORIS was 1.81 with a standard 
error of .15. This is an indication that they have a strong sense of operational 
satisfaction and view the impacts of changes to their program as important. 

• The Mean rating from those surveyed for CNSF Web was 3 with a standard 
error of .19. This is an indication that they have a moderate view of impacts 
on their program. 


Results from ASC Questions 

3.5 
3 

a 2.5 

'ro 2 
cc 

S' 1.5 


0.5 

0 


Figure 12. Survey Results for five Acquisition Strategy Characteristics (ASC) 

The participants also answered questions regarding the level of 
effort it took to bring their programs into existence, as displayed in Figure 13. Most 
programs were implemented in two years. CV SHARP has taken five years; according to 
the PM they are still developing new usage features. SHARP took two years for the initial 
implementation and is planned out for the next 10 years. Integrating these programs 
within their current baselines varied. Even though the Certification was achieved by most 
in one year, most agree that it a task that must be managed well. 


-I 



CNSF Web TORIS SHARP CV SHARP 

Programs Reviewed 
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Establishing Program Operations 



■ ■ ■ ■ 


CNSFWeb TORIS SHARP CVSHARP 


■ Implement 

■ Integrate 

■ Certification 


Programs 


Figure 13. Survey Results for Time in Years for Program Development and Operation 

Next, the partieipants responded to questions regarding the level of 
difficulty experienced with the integration, procurement, and certification of their 
program, as displayed in Figure 14. They rated their experience on a scale of one to 
seven, with seven being the worst. The CV SHARP PM viewed all three categories as 
being very difficult. Across the board, program certification was challenging because the 
organizations did not have a trained and knowledgeable person to perform this task. 
Despite the fact that these programs are the product of a study or a critical mission need, 
procurement challenges still existed. 
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Program Development 

8 -- 



CNSFWeb TORIS SHARP CVSHARP 


■ Integration 

■ Procurement 

■ Certification 


Programs 


Figure 14. Level of Difficulty for each program’s creation 


(2) Survey Responses - Distribution by Program. The set of 
charts in Figure 15 provide a depiction of how the replies to the survey are distributed by 
program. CNSF Web rated the survey questions has having moderate to little impact on 
their program, which shows confidence in their Web based program to operate well in 
spite of changes. TORIS and CV SHARP rating shows their concern regarding external 
factors affecting their program’s operation and growth, which shows their sensitivity to 
changes in their customer, policy, and requirements. The SHARP rating, which mostly 
falls between 1 and 3, indicates confidence in their stability to maintain operations 
despite changes. 
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CNSF Web 


TORIS 


15 

10 

5 

0 



1 2 3 4 5 More 



Question Response Distribution 


Question Response Distribution 


SHARP 


CV SHARP 


More 



Question Response Distribution 


Question Response Distribution 


Figure 15. Program response to Survey Questions 


b. Data Variability Analysis 

The mean data and the standard deviation, of the independent programs, is 
analyzed in terms of variability when compared with the ASCs shown in Table 5 
(Rumsey, 2003). CNSF Web and SHARP show zero variability, or no deviation, for 
Resource Balance, This indicates the participants replied exactly the same way to the 
questions. Additionally, TORIS and CV SHARP show a very low variance for Resource 
Balance. The participants replied to this line of questioning with very similar responses 
regarding its importance and significance. All four programs show a moderately low 
variance for flexibility. This is an indication that they are similar in their thinking on how 
to deal with change. Risk management and Stability show the highest levels of variability 
across all four programs, this is an indication that training in the importance of risk 
managements and guidance on increasing stability may need to be a future focus as 
organizations seek out improvement measures. 
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Variable 

CNSF 

Web 

TORIS 

SHARP 

CV 

SHARP 

Realism 

3.6(1.1) 

2(.7) 

2.2(.8) 

1.4(.5) 

Stability 

3(1.2) 

1.9(.9) 

2.3(1.1) 

2.4(1.5) 

Resource 

Balance 

2(0) 

1.3(.6) 

2(0) 

1.7(.6) 

Flexibility 

3.3(.5) 

1.8(.5) 

2.5(.6) 

2(.8) 

Risk 

Management 

2.9(1) 

1.9(1) 

2.4(1.2) 

2(1.4) 


Table 5. Mean (Standard Deviation) - Programs vs. ASC 


c. Nonparametric Measuring Techniques 

The focus of this section is to compare the responses from the four 
programs. The population is small in order to establish a comparison between programs 
with similar missions and comparable dollar values. Small populations are better suited 
for nonparametric evaluations techniques. These techniques, also called distribution-free 
statistics, focus on ranked data. These techniques also look to determine if the population 
locations are different, unlike the parametric measures that determine if the mean is 
different (Keller, 2009). They allow us to infer something meaningful beyond descriptive 
statistics (Schutte, 1977). The Kruskal-Wallis test will determine if the data meets the 
criteria for the hypotheses. Next, the Wilcoxin Rank Sum test will compare those groups 
that fall outside of the hypothesis acceptance level. Finally, the Spearman Correlation 
examines the relationship between two chosen variables. 

(1) Kruskal-Wallis Test. The Kruskal-Wallis is a “powerful 
and attractive nonparametric test for analyzing independent samples (Teng, 1978).” The 
Kruskal-Wallis tests how each program’s rankings faired against each other per ASC. 
The Significance level was .10. The P-Value is the focus for accepting or rejecting the 
null (HO) hypotheis. Therefore, if the P-Value is less than .10, then the HO is rejected. 
The null (HO) and alternative (HA) hypothesis are as follows: 
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• HO: The location of all populations are the same 

• HA: At least two population locations are different 


A numerical sample was established for each program based upon 
a ranking order. The new samples for each program underwent the Kruskal-Wallis test 
based on the ASC’s. Of the five tests conducted Realism and Flexibility showed they 
were significantly different because they rejected the HO. Both Realism and Flexibility 
have P-Values less than the .1 significance level with a value of .07 and .06, respectively 
as shown in Figure 16. The comparisons that show significance is highlighted. While 
the Kruskal-Wallis test can determine if there is a difference, it cannot predict which area 
that caused the actual difference to occur. The Wilcoxon-Rank Sum test can determine 
the actual area of the difference (Keller, 2009). 


Realism 



Stability 



Resource Balance 



Kruskal-Wallis Test 



Kruskal-Wallis Test 



Kruskal-Wallis Test 


1 

Group 

Rank Sum 

Observations 

Group 

Rank Sum 

Observations 

Group Rank Sum 

1 

Observatior 

CNSF Web 

76.5 

5 


CNSF 

131 

7 


CNSF 

24 

3 

TORIS 

52.5 

5 


TORIS 

78 

7 


TORIS 

12 

3 

SHARP 

54 

5 


SHARP 

99 

7 


SHARP 

24 

3 

CVSHARP 

27 

5 


CV SHARP 

98 

7 


CV SHARP 

18 

3 

H Stat 


7.02 


HStat 


3.0422 


H Stat 


2.5385 

df 


3 


df 


3 


df 


3 

p-value 


0.0713 


p-value 


0.3852 


p-value 


0.4684 

chi-squared Critical 

7.8147 


chi-squared Critical 

7.8147 


chi-squared Critical 


7.8147 

Flexibility 

Risk Managerrent 





Kruskal-Wallis Test 



Kruskal-Wallis Test 





Group 

Rank Sum 

Observations 

Group 

Rank Sum 

Observations 




CNSF 

53.5 

4 


CNSF 

176 

8 




TORIS 

19.5 

4 


TORIS 

106 

8 




SHARP 

37 

4 


SHARP 

139 

8 




CVSHARP 

26 

4 


CV Sharp 

107 

8 




HStat 


7.318 


H Stat 


4.6676 




df 


3 


df 


3 




p-value 


0.0624 


p-value 


0.1978 




chi-squared Critical 

7.8147 


chi-squared Critical 

7.8147 





Figure 16. Kruskal-Wallis Test Results 
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(2) Wilcoxon Rank Sum Test. The Wilcoxon Rank Sum Test 
will compare the individual components for each independent group (Keller, 2009). The 
results from the Kmskal-Wallis Test indicate that both Realism and Flexibility were 
significantly different. The Wilcoxon Rank Sum Test will examine each program under 
the two variables to determine which areas caused the difference. The significance level 
was set at .10 and the null and alternative hypotheses are below: 

• HO: The two population locations are the same. 

• HA: The two populations are different (using the two sided HA). 

(a) Realism. Figure 17 shows all of the Wilcoxon 
comparisons conducted for Realism and those with a significant difference are 
highlighted. Three of the six comparisons under realism showed a significant difference. 
CNSF Web reported their program as significantly more realistic than TORIS, SHARP, 
and CV SHARP. The P-values range from .02 to .08. Their rank sum comparisons are as 
follows: 

• CNSF Web - 37 > TORIS - 18 

• CNSF Web - 36 > SHARP - 19 

• CNSF Web - 39 > CV SHARP - 16 
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Wilcoxon Rank Sum Test 

Wilcoxon Rank Sum Test 


Rank Sum Observations 


Rank Sum Observations 

CNSfWeb 

37 5 

CNSFWEB 

36 5 

TORIS 

18 5 

SHARP 

19 5 

z Stat 

1.9845 

z Stat 

1.7756 

P(Z<=z) one-tail 

0.0236 

P(Z<=z) one-tail 

0.0379 

z Critical one-tail 

1.2816 

z Critical one-tail 

1.2816 

P(Z<=z) two-tail 

0.0472 

P(Z<=z) two-tail 

0.0758 

z Critical two-tail 

1.6449 

z Critical two-tail 

1.6449 


Wilcoxon Rank Sum Test 


Wilcoxon Rank Sum Test 


Rank Sum 

Observations 


Rank Sum Observations 

TORIS 

33.5 

5 

SHARP 

34.5 5 

CVSHARP 

21.5 

5 

CVSHARP 

20.5 5 

z Stat 

1.2534 


z stat 

1.4623 

P(Z<=z) one-tail 

0.105 


P{Z<=z) one-tail 

0.0718 

z Critical one-tail 

1.2816 


z Critical one-tail 

1.2816 

P{Z<=z) two-tail 

0.21 


P(Z<=z) two-tail 

0.1436 

z Critical two-tail 

1.6449 


z Critical two-tail 

1.6449 

Wilcoxon Rank Sum Test 


Wilcoxon Rank Sum Test 


Rank Sum 

Observations 


Rank Sum Observations 

CNSFWEB 

39 

5 

TORIS 

25.5 5 

CVSHARP 

16 

5 

SHARP 

29.5 5 

z Stat 

2.4023 


z Stat 

-0.4178 

P{Z<=z) one-tail 

0.0081 


P(Z<=z) one-tail 

0.3381 

z Critical one-tail 

1.2816 


z Critical one-tail 

1.2816 

P{Z<=z) two-tail 

0.0162 


P(Z<=z) two-tail 

0.6762 

z Critical two-tail 

1.6449 


z Critical two-tail 

1.6449 


Figure 17. Wilcoxon Rank Sum Test for Realism 


(b) Flexibility. Figure 18 shows all of the comparisons 
conducted by the Willcoxon for Flexibility, with those showing a significant difference 
highlighted. Two of the six comparisons under flexibility showed a significant difference. 
Of the two comparisons CNSF Web rated their system as more flexible than TORIS or 
SHARP rated their system. The P-values range from .02 to .06. Their rank sum 
comparisons are as follows: 
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• CNSF Web - 26 > TORIS - 10 

• CNSF Web - 24.5 > CV SHARP - 11.5 


Wilcoxon Rank Sum Test 


Wilcoxon Rank Sum Test 



Rank Sum 

Observations 


Rank Sum Observations 

CNSFWEB 

26 

4 

CNSFWEB 

23 4 

TORIS 

10 

4 

SHARP 

13 4 

z Stat 

2.3094 


z Stat 

1.4434 

P(Z<=z) one-tail 

0.0105 


P(Z<=z) one-tail 

0.0745 

z Critical one-tail 

1.2816 


z Critical one-tail 

1.2816 

P(Z<=z) two-tail 

0.021 


P(Z<=z) two-tail 

0.149 

z Critical two-tail 

1.6449 


z Critical two-tail 

1.6449 


Wilcoxon Rank Sum Test 

Wilcoxon Rank Sum Test 



Rank Sum Observations 


Rank Sum 

Observations 

TORIS 

13 4 

TORIS 

16.5 

4 

SHARP 

23 4 

CVSHARP 

19.5 

4 

z Stat 

-1.4434 

z Stat 

-0.433 


P(Z<=z) one-tail 

0.0745 

P(Z<=z) one-tail 

0.3325 


z Critical one-tail 

1.2816 

z Critical one-tail 

1.2816 


P(Z<=z) two-tail 

0.149 

P(Z<=z) two-tail 

0.665 


z Critical two-tail 

1.6449 

z Critical two-tail 

1.6449 


Wilcoxon Rank Sum Test 

Wilcoxon Rank Sum Test 



Rank Sum Observations 


Rank Sum 

Observations 

SHARP 

21 4 

CNSFWEB 

24.5 

4 

CVSHARP 

15 4 

CVSHARP 

11.5 

4 

z Stat 

0.866 

z Stat 

1.8764 


P(Z<=z) one-tail 

0.1932 

P(Z<=z) one-tail 

0.0303 


z Critical one-tail 

1.2816 

z Critical one-tail 

1.2816 


P(Z<=z) two-tail 

0.3864 

P(Z<=z) two-tail 

0.0606 


z Critical two-tail 

1.6449 

z Critical two-tail 

1.6449 



Figure 18. Wilcoxon Rank Sum Test for Flexibility 


(3) Spearman Rank Correlation. The Spearman Rank Correlation 
analyzes rank data. It measures how closely two sets of rankings agree with each other 
(Schutte, 1977). The results from this test are displayed below in Table 6. The closer the 

value is to one indicates how well the variables agree with one another. Since most of the 
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variables for this study are hardly related, it shows the diversified nature of the variables. 
The variables were used to evaluate the mindset involved in the development and 
operation of eaeh program. This test adds validity to the evaluation variables because 
they cover different aspects of the programs. 



Realism 

Stability 

Resource 

Balance 

Flexibility 

Risk 

Management 

Realism 

1 

-0.01 

-0.06 

0.32 

0.27 

Stability 

-0.01 

1 

0.29 

-0.09 

-0.25 

Resource 

Balance 

-0.06 

0.29 

1 

0.15 

-0.21 

Flexibility 

0.32 

-0.09 

0.15 

1 

0.41 

Risk 

Management 

0.27 

-0.25 

-0.21 

0.41 

1 


Table 6. Spearman Correlation Test 


3. Results of Quantitative Analysis and Their Applicability to the 
Organization 

a. Realism 

(1) Survey results. Realism is the ability to set obtainable goals. 
From the analysis, three areas present feedback on how realism is addressed. First, 
Figure 15 displays how the Kruskal-Wallis Test showed realism was significantly 
different because it rejected the HO. This indicated that these results did not come from 
a random chance of occurrence. Secondly, Figure 16 depicts how the Wilcoxon- Rank 
Sum tests the way each organization responded to the realism questions. Three areas 
showed a significant difference. CNSF Web reported their program as significantly more 
realistic than TORIS and CV SHARP. Third, Figure 13 showed how the lack of a 
knowledgeable expert made certification difficult, indicating a need for a commodity 
manager or team of experts to oversee technological changes. 
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(2) Organizational patterns. The programs examined in this study 
showed similar behaviors that realism teehniques ean improve. First, all of the programs 
have a ehange management approval proeess, but it resides at the program level. 
Deeisions at the program level should fit into the organization’s goals for a funding 
strategy. Additionally, due to the requirements-driven nature of the programs, it is 
diffieult to establish goals for an operational baseline. Establishing a baseline will 
provide a eore proeess that must be funded from year to year. Third, future program 
vision is needed in order to ereate version eontrol. The plan for future versions assist in 
funding request justifieations and provides a visual that shows how a program evolves 
from a basie eapability to a more responsive and robust program. 

b. Stability 

(1) Survey results. Stability exists when an organization ean 
overeome negative disruptions that affeet eost, sehedule, and performanee requirements. 
The results from the mean of the overall survey. Table 5, revealed that SHARP was most 
often rated between 1 and 3. This rating indieated Program Managers’ eonfidenee in 
their ability to maintain stable operations in the midst of ehanges. In Table 5, the mean 
and the standard deviation eomparison show a high variability in stability for all 
programs. This indicates that guidance in increasing stability may need to be a future 
focus. 

(2) Organizational patterns. With the programs in this study, 
variability could be high since most of them are in a continuous maintenance mode that 
consists of adding new capability while trying to resolve current problems. This shows 
that the programs are viable and in a responsive mode to meet operational requirements. 
The underlining thought is that there is no hard reference point to where the programs 
started, currently reside, or future capability. 

c. Resource Balance 

(1) Survey Results. Resource balance accounts for the use of 
resources to meet requirements. Early in the analysis, Eigure 13 depicted the 
procurement challenges with each program. These programs developed out of an 
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operational need or a directed study. Their existence has already proven to be worthy; 
therefore, a sustaining funding vehicle is needed. Table 5, the mean and standard 
deviation analysis, showed zero variability for CNSF WEB and SHARP for Resource 
Balance, indicating the participants rated the balance for resources and requirements as 
significant and important. 

(2) Organizational Patterns. Observations of the organizations 
show similar behaviors. Most seem to operate from a single funding line which is good 
from a control perspective. But when funding level reductions are mandated, it would be 
helpful to have an idea of what services could be discontinued with little impact on 
organizational performance. Additionally, the funding vision for how to best leverage the 
funds for future efficiencies seem to be short term. Another observation is that managers 
that control a single funding line seem to be reactive, versus proactive, to environmental 
changes. These challenges emerge out of the concern for changes in the funding levels. 
A dedicated funding vehicle could address these challenges. A funding plan will not 
make the problems disappear, but it could provide leverage for decision making over the 
long term. 


d. Flexibility 

(1) Survey Results. Flexibility consists of the ease with which 
changes and failures can be accommodated without significant changes in resources 
requirements. Figure 13 shows ratings for both TORIS and CVSHARP, indicating 
concern for external factors affecting their program’s operation and growth by showing 
concern for changes in their customer, policy, and requirements. Figure 16 shows the 
Wilcoxin-Rank Sum tests results for the way the participants responded to the Flexibility 
questions. Two comparisons showed a significant difference. CNSF Web rated their 
system as more flexible than TORIS or SHARP rated their system. 

(2) Organizational patterns. Flexibility focuses on how resources 
are used to meet requirements. Changes are made as requirements develop and funding 
is not always considered. TORIS users pay into the program by being taxed. So when a 
requirement emerged, there was no additional funding. SHARP has an annual budget for 

current operations. Their users fund any new capabilities. 
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e. Risk Management 

(1) Survey Results. Risk Management centers on proactively 
determining ways to mitigate risk from external and internal factors that affect cost, 
schedule, and performance. In Figure 13, CNSF Web rated their program as having 
moderate to little impact from risk factors. This shows confidence in their Web based 
program to operate well in spite of changes. In Table 5, Risk Management showed a high 
level of variability for all programs. This is an indication that training on the importance 
of risk management should be a future focus. 

(2) Organizational Patterns. Variability in risk management 
indicates that those surveyed do not view it the same way. The idea of managing risk is 
to identify the risk and set measures in place to reduce the impact of their occurrence. 
The programs are managed with similar behaviors. Most programs are using the same 
contractor or have used one contractor for the majority of their programs existence. This 
could be good because the contractor would have an understanding of the program from 
the beginning. The risk worth identifying is what will happen if this contractor is 
replaced. To mitigate this risk, the organization should take actions similar to the 
following: 

• Obtain and update current documentation detailing how the program 
operates, where the data is stored, and data retrieval steps 

• Have their contractors sign statement of nondisclosure for information 
critical to the organization. 

• Have some government employees trained on how the program works 

• Create metrics for measuring both the contractor and the programs’ 
performance. 

/. Topics Applicable to All ASCs 

(1) Survey Results. Some of the analysis presented overarching 
ideas. In Figure II, The Mean rating from those surveyed for TORIS was 1.81 with a 
standard error of .15. This is an indication that they have a strong sense of operational 
satisfaction and view the impacts of changes to their program as important. While from 
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the same figure, the mean rating from those surveyed for CNSF Web was 3 with a 
standard error of .19. This is an indication that they have a moderate view of impacts on 
their program. Finally, in Table 6 the Spearman Correlations indicated that most of the 
variables for this study are hardly related. This is significant because it shows the 
diversified nature of the variables used to evaluate these programs. This ultimately gives 
credibility to the overall study for reviewing these programs from different perspectives. 

(2) Organizational Patterns. Overall, the goals of the programs 
are to provide accurate, timely, and widely assessable proficiency statuses into the 
DRRS-N. The program managers have a heightened sense of concern for satisfying user 
requirements, meeting customer needs, and reporting the status of the war fighting 
equipment. 

C. QUALITATIVE ANALYSIS OF CNSF AND CNAF ACQUISITION 

STRUCTURE AND DECISIONS 

I. Comparison of Commander, Naval Surface Force (CNSF) and 
Commander, Naval Air Forces (CNAF) Organizational Values on IS 
Acquisition decision making 

Understanding an organization’s culture and priorities is critical to the successful 
implementation of any strategic plan or change initiative. To begin the strategic planning 
process, an organization must assess its values, core processes, and goals (Fisher, 2010). 
The CNSF mission is to support the SWE in its effort to provide “warships ready for 
tasking.” Therefore, its core mission areas deal with promoting and tracking superior 
operational readiness of the fleet at all times. The Navy’s progression toward warfare 
enterprises is centered on the concept of alignment with the purpose to provide 
continuous mission capability at lower cost. The Type Commander (TYCOM), 
Supporters, and Providers are aligned to meet the tasking requirements for the Enterprise. 
Through superior training and readiness CNSE strives to “provide operational 
commanders with well-trained, highly effective, and technologically superior surface 
ships and Sailors” (Surface Warfare Enterprise, 2010). However, CNSE also states that 
in order to obtain this goal, a “significant level of adaptability and flexibility” is required. 
CNSE is a mission-focused organization whose strategic approach to acquisition of 
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Information Systems (IS) must be flexible and adaptable to dynamic requirements and 
evolving mission needs. Based on conversations with CNSF program managers, 
processes that are at the core of CNSF business are: 

• Ensuring operational readiness of surface forces is met 

• Ensuring manning requirements of surface forces are met 

• Ensuring surface forces have the latest and most effective technology 

• Tracking and Reporting operational training and readiness 

• Collaboration with various Elect Readiness Enterprise (ERE) components 

Although CNSE Information Systems must be adaptable and flexible, the 
processes that are core to completing its mission remain static. The command requires 
adaptable and flexible systems that will allow them to execute their mission efficiently 
and effectively. However, it is important to note, while IS are vital to CNSE executing 
core mission areas, IS decision making is not a high level priority in the command’s 
strategic vision. This is largely due to the command’s lack of IS strategy, nor a strategic 
vision for the acquisition of systems and services needed to maintain their current IS 
posture or make future improvements. 

Currently, CNSE is annually funding $2.2M for Training and Operational 
Readiness Information Service (TORIS) from monies allocated for ships’ operations 
(Ship Ops). Ship Ops money is redirected from every ship in the surface force to fund 
TORIS upgrades, development, and maintenance. As a result, ship Commanding 
Officers (COs) may have to deal with budgeting shortfalls in various areas during daily 
shipboard operations. Eor example, when ships need repair parts or consumable items 
such as a motor for a refrigeration system or toilet paper the CO may be forced to wait for 
that vital repair part or redirect funding from other area of his Operational Target 
(OPTAR). Since money for funding the operational units is targeted first when funding 
TORIS contracts, surface units are forced to maintain their level of readiness with fewer 
resources. Although TORIS is a vital program to the operational commander, the 
program and the surface forces may be better served if the funding for the system was its 
own budget line item. 
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While the TORIS program was initially designed to track and report readiness of 
the surface fleet, it has proven to have useful capabilities far beyond its original purpose. 
TORIS not only provides contracted Analysts with vital metrics on fleet readiness, it also 
automates routine administrative functions for the ATG command, such as electronic 
mustering, automating sea pay activities for shipboard trainers, and electronic tracking of 
administrative correspondence. These added capabilities reveal the value of the system, 
and foreshadow of usefulness that is yet to be realized. TORIS output data has been 
requested by organizations other than those within the SWE. According to TORIS 
Program Contractors, the Navy Staff (OPNAV) and the US Coast Guard (USCG) also 
use TORIS to supplement their own awareness on various action items within their 
organizations. 

TORIS was initially acquired under an evolutionary acquisition strategy and 
developed using a spiral method. This method was chosen because the requirements of 
the system were not clearly stated and the end-state of the system was unknown. 
According to one of the original TORIS developers, “TORIS was implemented as a cost 
savings measure. When the acquisition strategy was designed we [TORIS Development 
Team] did not know what we did not know. The problem was not well defined. 
[Therefore], developing with the philosophy of maintaining agility in both programming 
and project management approach, we were able to produce TORIS as a global system 
deployed to every ship for a fraction of the cost of any other comparable system.” This 
approach yielded a quality product that has met its goals of cost-savings. The system 
continues to use this approach of spiral development to achieve further capability to meet 
continually evolving user requirements. 

Changes to the TORIS configuration are vetted through a Configuration Control 
Board (CCB), which oversees and approves changes in the various data views and 
functionality of the system. The CCB is made up of the Afloat Training Group (ATG) 
Commodores and N7 (Training and Readiness Department Heads) from the various ATG 
commands in the fleet. This board meets periodically to discuss new requirements for 
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training and readiness for the fleet. The requirements from the CCB are passed on to the 
system developers who add the coding necessary to achieve the desired system 
functionality. 

Collaboration is vital to CNSF support of the SWE. CNSF Web allows for CNSF 
to disseminate policy documentation to surface units. SURFORWEB also serves as a 
central location for knowledge management resources critical to successful execution of 
shipboard missions. SURFORWEB serves as a centralized information repository for 
units to access. However, it is important to note that CNSF Web is not a critical system 
for ships to get underway and execute mission requirements. Although the program 
supports the SWE vision of collaboration and alignment throughout the enterprise, there 
is no financial data to support a cost-benefit analysis for determination of a return on 
investment (ROI). This presents an issue for the command when faced with decisions of 
where to cut costs and decrease or increase spending on IS assets. Previous iterations of 
SURFORWEB supported staffs and infrastructure on both coasts at Commander, Naval 
Surface Forces Pacific (CNSP) and Commander, Naval Surface Forces Atlantic (CNSF) 
under multiple contracts. Recently these contracts were migrated under one umbrella 
contract supporting SURFORWEB manning and infrastructure at CNSF. 

CNSF acquisition of the TORIS and SURFORWEB programs was a result of 
organizational priorities of mission accomplishment, through enterprise collaboration and 
alignment. However, decision makers did not fully consider what cost efficiencies, if 
any, would be provided by these systems. Neither system has cited any key performance 
parameters for cost savings. The lack of considering cost saving when acquiring fails to 
leverage the cost benefits of an enterprise organizational strategy. CNSF desires systems 
that will support their mission requirements while maintaining a level of adaptability for 
future capability as indicated by the dynamic state of the TORIS program. While 
SURFORWEB contribution to CNSF mission is evident, its return on investment, if any, 
has yet to be discovered. 
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a. 


CNAF IS Programs 


The SHARP program tracks readiness of the aviation force down to 
individual level. Commander, Naval Air Forces (CNAF) is tasked with monitoring 
readiness of both the carrier fleet and the individuals that make up the squadrons’ flight 
crews. The SHARP program has utilized the same contractor since its inception. The 
value that SHARP adds to CNAF is considerable. The TYCOM has a view into the 
Navy’s air force enterprise down to the individual level so that it may tailor training 
requirements to provide the greatest benefit to the aircrews that make up the various 
squadrons throughout the fleet. 

Much like the TORIS program, the Carrier Sierra Hotel Aviation 
Readiness Program (CV SHARP) tracks training and operational readiness of the Carrier 
fleet. CV SHARP is an evolutionary acquired program that has been developed through 
increments. Since its initial development CV SHARP has been a funding priority for 
CNAF. However, similar to CNSF, CNAF funding for CV SHARP comes from monies 
allocated for the Flight Hours Program (FHP). Similar to Ship’s Operating (Ships Ops) 
funds, FHP funds pay for the numerous aspects associated with flight operations (i.e., 
maintenance, fueling, training, etc.,) CNAF priorities are similar to those of CNSF. 
Supporting the NAE through training and operational readiness of all naval aviation 
assets is top priority. CV SHARP uses a Change Control Board (CCB) to vet functional 
changes within the programs. However, The Program Manager (PM) has final decision 
authority before any changes can be made to the system. According to the PM, “All 
software development requirements are written in a Customer Acceptance Document, 
which lists all of the requirements for that iteration and the timeline associated with it” 
(CV SHARP Program Manager, 2010). The CCB allows for version control of the 
program aids in prohibiting increasing program requirements and avoiding budget 
overruns. Research into the CV SHARP system acquisition did not yield indications of 
cost-savings metrics or key performance parameters for savings. 
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2. Merging Acquisition Strategies with CNSF Organizational Values 

CNSF requires Information Systems (IS) that support the readiness of the 
operational forces and support SWE requirements. Acquisition of these IS systems must 
converge with organizational values to ensure a seamless integration of the system into 
the organization, and that the greatest value is being attained through the systems use. 
This section will discuss how various IS acquisition strategies can be used to incorporate 
CNSF organizational priorities. While there will not be a single strategy that will address 
all organizational priorities, there are benefits inherent in each strategy that can best 
achieve certain command priorities. 

a. Evolutionary Acquisition Strategy (EAS) 

A key factor when considering an EAS is the reduction of time it takes to 
deliver an initial operating capability (IOC) to the fleet. For a command tasked with 
facilitating and maintaining operational readiness, reduced cycle time should be a 
priority. Reduced cycle time is due in part, to leveraging mature technologies either from 
the private or government sector. SURFORWEB is built on the Microsoft SharePoint 
software program. The benefit of enhanced granularity of requirements inherent within 
the EAS can provide a benefit to organizations that require information systems but are 
uncertain of what their needs are. The incremental development process used in the EAS 
provides the flexibility that CNSF needs to meet dynamic capability requirements. 
Subsequent versions of the IS can include configuration changes. 

b. Product Line Acquisition 

CNSF is a geographically dispersed organization whose IS requirements 

to execute its core mission processes may be well served using PEA. The core asset 

development function, as discussed in chapter two, inherent within the PEA strategy 

would prove beneficial to an organization whose mission is top priority. Ultimately, a 

product could be developed as a result of core asset development and management. 

Currently, the CNSF IS architecture is not well defined and lacks a strategic vision. PEA 

offers the benefits of developing products that can seamlessly integrate into a well- 

defined architecture. Additionally, Software reuse from proven systems can offer a cost 
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savings in time and value for the command. One of the cost savings initiatives of TORIS 
was its capability to allow for manning decreases of up to 40 percent. Greater efficiency 
in using human resources is a benefit of PLA as discussed in chapter two. This approach 
would require the command to execute a detailed assessment of their Information System 
priorities, and determine how these priorities and there is needs will fit to enhance core 
processes, and provide capability for future process changes. 

c. Performance Based Services Acquisition (PBSA) 

A primary benefit of PBSA is improved quality and better value. An IS 
that provides value to the command’s mission is an organizational priority for CNSF. 
Therefore instituting an acquisition strategy that requires high quality performance from 
contractors in developing those systems would offer benefit to the command. This 
strategy would require clear requirements in order to facilitate a clear statement of work 
(SOW). CNSF requires measurable outcomes of contractor and system performance to 
assist in making spending decisions. A benefit of PBSA is that contractors are measured 
according to value criteria that the command determines. In this strategy a cost savings is 
realized through competition and innovation. CNSF would simply provide system 
performance requirements and the contractor would assume the risk in meeting them. 
Figure 19 is a re-creation from the Bridge Technology Corporation training on PBSA, 
displays the relationship between the various components in PBSA. 


72 



Clear 

SOW 


PBSA 


CUSTOMER/ 

CONTRAQOR 

RELATIONSHIP 


MEASURABLE 

OUTCOMES 


Figure 19. Relationship of components in PBSA (After PBSA Training, Berendt, 2005) 

The greatest benefits for a command such as CNSF, are the measurable outcomes. 
SURFORWEB currently has no performance outcome to measure whether all objectives 
are being met, or that contractor performance is at some benchmark of acceptance. 

3. GAO Case Study of Commercial Transformation in Decision Making 

on IS Acquisition Spending (Case Study Located in Appendix C) 

a. Introduction 

The government Accountability Office (GAO) conducted a study of six 
companies from the commercial industry in order to capture lessons learned on how they 
evolved from traditional to strategic service acquisitions. Strategic decision making was 
based upon four concepts: 

• Any change effort must have top-level leader support 

• The organization should obtain financial knowledge through conducting 
a detailed spend analysis 
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• An organization must create the structure, processes, and roles for 
strategic decision making 

• There must be sustained leader support, open communication channels, 
and credible metrics for compliance validation 

By embracing these four concepts, the organizations in this study were 
able to better leverage their resources, reduce their operating costs, better manage their 
service providers, and gain better quality from services rendered. 

Early in the process, the companies realized that they lack money flow 
control for acquiring services. Additionally, decision making for the acquisition of 
services was left to individual units such as finance, engineering, human resources, and 
maintenance, without top-level intervention or expert evaluation which hindered the 
ability to make coordinated decisions across the organization. Additionally, there were 
no tools to measure the services’ ability to meet company needs. Four general principles 
emerged as the most critical for achieving a strategic approach to purchasing services: 

• Secure up front commitment from top leaders 

• Obtain improved knowledge on service spending 

• Create supporting structure, processes, and roles 

• Enable success through sustained leadership, communication, and 
metrics 

(1) Senior Eeadership Involvement. Senior leadership 
involvement from the beginning was paramount to the success of the change initiative. 
The leaders provided direction, vision, and supervised the development of a common 
process approach. In some cases, leaders associated their name to new ventures in order 
to secure buy-in. 

The shift from a traditional to strategic decision-making approach 
required active participation from senior leaders. For example, traditionally, services 
belonged to one business section, while strategically they are viewed centrally. 
Examining how change occurs; traditional managers are not active participants. But, 
strategically, they provide direction and vision, by providing goals, targets, and feedback 
on achieving them. 
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It is important to have senior leaders spearheading reengineering 
efforts because they have the authority to direct support, the responsibility to set 
corporate agenda, and the power to remove barriers that block change. Previous research 
has documented that the failure of reengineering efforts are largely attributed to the lack 
of top management commitment. 

(2) Spending Analysis. To gain knowledge on current trends in IS 
spending; a spending analysis should be conducted. In this case study the spending 
analysis was initially used to determine how much was being spent for services and to 
whom was it paid. The analysis revealed that the companies in the study were buying 
similar services from numerous providers, at often times varying prices. To make 
strategic decisions, they would have to develop a deliberate approach to planning and 
managing their acquisitions. 

Strategically approaching acquisition decision making focused on 
developing credible, reliable, and timely data on services acquired. Strategically, data is 
used to find opportunities that will rationalize the supplier base and reduce costs, instead 
of being used as an after-the-fact reporting tool. A spend analysis should identify the 
following: 

• What types of services are being acquired? 

• How many suppliers for a specific service the company is 
using? 

• How much they are spending for that service, in total and 
with each supplier? 

• Which units within the company are purchasing the 
services? 

(3) Structure, Processes, and Roles. To achieve an enterprise 
perspective, the companies revamped their procedures for acquiring services in terms of 
their structure, processes, and roles. Initially, their traditional structure hindered their 
ability to coordinate across their organizations. To alleviate this obstacle the companies 
changed their structures. They made three major changes: 
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• Elevated and/or expanded the role of the company’s procurement 
organization. 

• Established cross-functional teams with a mix of knowledge, 
expertise, and credibility. 

• Established dedicated commodity managers to oversee key 
services. 

The implemented changes evolved their role from one of 
resourcing to that of advisory. 

(4) Sustained Eeadership, Communication, and Metrics. 
Resistance and cultural barriers hindered the companies’ reengineering efforts. Three 
critical elements for overcoming these challenges were identified. Eirst, leadership 
support must be sustained throughout the life of the project. Second, open 
communication lines must exist between the business units and the purchasing 
organization. Einally, credible metrics should be used to add validity to organizational 
decisions. 


b. Conclusion/Summary 

The companies in the GAO study recognized that there was a gap between 
their core business processes and their consideration of the IS as a core asset. This was 
evident at the amount of cash outflows they experienced for duplicated services. DoD 
components such as CNSE are at a similar crossroads, wherein it is necessary for 
organizations like CNSE to take a more strategic approach toward spending in the realm 
of IS. Gaining greater knowledge on how monies are spent in IS acquisition will fuel 
more strategic thinking for acquisition decision makers. A successful change initiative 
must begin from the top and be pushed down to the subordinate organization. This top- 
level leadership support must be maintained throughout the transformation. 

4. Findings on CNSF Shortfalls in Strategic IS Acquisition 
a. Lack of Strategic IS vision 

In this chapter, it has been stated that CNSF does not have a strategic 

vision of its information systems infrastructure. This lack of an IS strategy causes a gap 

in the command’s ability to make meaningful decisions for spending to better leverage its 
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IS capability. CNSF must identify how IS fits within its core business processes. 
Additionally, the core processes must be prioritized to match organizational values and 
goals before the command can strategically approach the acquisition of further IS 
services and capabilities. Although CNSF has been successful at utilizing IS to complete 
its mission requirements it is incurring greater costs in order to do so. The possibility of 
looming budget cuts is driving the command to give greater consideration to IS spending. 
Ultimately the goal is to maintain current mission capability at a lower cost. A robust 
and detailed IS vision will provide the framework for CNSF IS infrastructure. It will 
provide organizational decision makers with a frame to understand what is being acquired 
and how it fits in the organization’s overall mission. Ultimately, enabling CNSF to better 
support the SWE. 

b. IS is a Mission Enabler Not a Priority 

IS, such as TORIS and SURFORWEB, are essential applications to 
effective mission accomplishment and cost-savings for CNSE. However, the command 
did not begin to consider them as such until spending trends in the IS realm began to 
make a noticeable increase. Much like CNAE and the companies discussed in the case 
study, CNSE views IS as a mission enabler and not central to the organization’s success. 
This is evident in the lack of an executive level IS advisor such as a Chief Information 
Officer (CIO). There is no top-level IS commodity executive with detailed knowledge of 
mission requirements, fiscal understanding, guided by a strategic IS vision to advise 
command leadership on how to best fund IS programs to most-effectively meet the 
mission goals, for the best value. 

c. IS Programs Lack Metrics for Return on Investment (ROI) 

The programs observed in this study were all developed to perform 
specific functions to meet CNSE mission requirements. While the programs are effective 
in executing these requirements, none were designed with output for ROI metrics in 
mind. CNSE wishes to analyze where its funding is best applied to better leverage its IS 
capability. However, without proper metrics the decision on where to increase or 
decrease funding is difficult at best. Therefore CNSE does not truly understand the value 
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that they are receiving from utilizing IS. There are currently no metrics or key 
performance factors for any of the systems reviewed in the study. These performance 
measurements also extend to the contractors tasked with developing and maintaining the 
systems. There is currently a lack of review of contractor performance to ensure quality 
and contract value. 

d. ASC Improvement Measures 

(1) Realism. Realism focuses on setting attainable goals (ASC 
document). Organizational goals need to align both operational and financial 
requirements. An option for achieving operational goal setting is through establishing a 
baseline that contains core processes. By establishing this baseline, fund managers can 
better earmark dollars for the daily mission. Additionally, by packaging new capabilities 
that can be accomplished in one to two years, funding can be allocated for that particular 
purpose. 

(2) Stability. Stability centers on overcoming negative 
influences and disruptions. Factors exist that work against achieving it and in support of 
it. Stability is best achieved through a dedicated structure that offers direction, advocacy, 
and commitment. Several options emerge. First, a CIO will provide direction and a focus 
agenda for achieving organizational goals. A CIO is only effective when the second 
point exist: top level commitment to the efficient use of IS resources. The leaders must be 
the voice in the command that requires compliance on the part of its employees. Third, 
an organization could also assemble a group of technological experts that understand 
technology requirements and funding constraints. 

(3) Resource Balance. The balance of resources against 
requirements can be achieved with internal and external techniques. An external 
technique could be by becoming a part of a larger buying program that allows them to 
purchase the same items but at smaller costs because they are purchasing along side of 
several other organization. Collectively they receive a discount because of quantity. 
This approach is used by the DON. The DON uses Department of Defense (DoD) 
Enterprise Software Initiative (DoD ESI) and the Eederal SmartBUY for commercial IS 

purchases (Enterprise Software Initiative, 2009). An internal approach could be through 
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conducting a spend analysis. This type of analysis will indicate what types of services are 
being acquired. It could also tell you how many suppliers exist for a specific service 
within the same company. It could also provide insight into how much is being spent for 
a service, a total amount, and to each supplier. Lastly, the test could also tell you which 
units within the organization are purchasing these services. 

(4) Flexibility. Flexibility is the organizations ability to 
accommodate changes and failures with minimal impact on resource requirements. A 
solution on how to achieve flexibility may reside in different areas. By establishing a 
baseline and conducting version control, the budget could be established to align with it. 
This helps to project funding requirements and increase the opportunity to be proactive in 
the anticipation of changes. Another option is to establish a funding line for core 
processes and share the financial burden of additional requirements. Lastly, establishing 
metrics for measuring the continued need for system capabilities is a way to keep the 
program viable and connected to its user community. 

(5) Risk Management. Risk management is based on 
mitigating risk that affects cost, schedule, and performance. Solutions that have already 
been mention can also serve here. Solutions like setting attainable goals, establishing a 
baseline, and incorporating version control can also work to mitigate risk. Another idea 
rests with measuring performance and knowing when milestones have been achieved. 
These measures empower the organization to be proactive in their decision making, an 
enhanced ability to manage provided services, and an audit trail for spending. 

D. SUMMARY 

This chapter presented qualitative and quantitative analysis of CNSF. The study 
presented a comparison of CNSF organizational values with a similar TYCOM CNAF. 
Ultimately mission accomplishment is CNSF main priority. However, since IS has not 
been placed as a priority the command has not instituted processes to identify metrics to 
provide ROI data required to begin discovering where to cut or increase spending within 
its IS infrastructure. A well-defined IS strategy, an executive level IS Advisor, and a 
thorough prioritization of organizational values with IS requirements are necessary to 

begin making lasting decisions concerning strategic IS acquisitions. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY OF RESEARCH 

The focus of this thesis was to examine CNSF current strategic approach to the 
acquisition of information systems (IS) and services. This research was the culmination 
of a 3-part project in the development of a cost effective information technology (IT) 
strategy for CNSF. Part of the CNSF system research was based on a baseline 
assessment of the command conducted by the first group in the study. This information 
included the functionality and background of the systems as well as the current command 
spending on system development and maintenance. The IS strategy that was discussed in 
chapter two was based on the IS strategy recommendation made by the second group. 
Enterprise Architecture was discussed to match the Navy’s current strategic approach to 
utilizing and leveraging IS to meet current mission needs. Chapter two focused on 
defining current acquisition strategies that are utilized by the DoD. Chapter three 
described CNSF systems, the organizational structure, and its current mission. 
Additionally, Commander, Naval Air Forces (CNAF) was introduced as a similar 
organization in order to provide a comparison of organizational values and approach to IS 
acquisition which was presented and discussed in chapter four. Chapter four presented 
both qualitative and quantitative analysis of the command’s values, and provided 
evaluation of the command’s shortfalls when considering the acquisition and spending on 
IS and services. These shortfalls provide areas of improvement for CNSF as it progresses 
toward a more strategic consideration of IS as a mission priority. 

This chapter will present conclusions based on the research and discussions with 
program decision makers and managers, as well as successful methods utilized in the 
acquisition of IS for similar organizations. This chapter will also make recommendations 
to assist CNSF decision makers in making improvements in the areas identified as 
shortfalls, and ultimately lead to achieving a more strategic approach to the acquisition of 
IS. 
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B. CONCLUSIONS 


1. Research Question Findings 

Which Strategic Approach to IS acquisitions broadly supports both CNSF 

missions and business practices? 

Since CNSF does not currently have an IS strategy that is paired with its 
organizational values and needs, it is difficult to identify a single acquisition strategy that 
will meet all of its objectives and requirements. However, in its support of the Surface 
Warfare Enterprise (SWE), which ultimately requires that “warships are ready for 
tasking,” any IS strategy can be instituted as long as it effectively supports that primary 
objective. As an organization, CNSE is taking the appropriate steps to becoming more 
aware of how to strategically acquire IS. According to the GAO case study discussed in 
Chapter four, commercial companies obtained greater knowledge and understanding of 
IS and IS spending on their way to achieving strategic IS acquisitions. Simply requesting 
that this three-part study be completed displays a shift to a more strategic view of the 
priority of IS within business processes and its acquisition. 

However, there are proven strategies that are utilized within the DoD that can be 
leveraged by CNSE to better provide a framework for acquisition decision making. 
According to the DoD 5000.2, Evolutionary Acquisition Strategy (EAS) has been touted 
as the acquisition approach of choice for the Department of Defense (DoD). EAS allows 
for incremental development of systems to meet the dynamic requirements inherent in the 
realm of warfare. Although CNSE responsibilities of manning, training, and equipping 
the surface forces are set, how these responsibilities are executed changes according to 
the need and threat. Within EAS, CNSE should utilize a Modular Open Systems 
Approach (MOSA) to allow future upgraded technologies to be integrated with current 
systems. Open systems utilize open standards that are accepted throughout commercial 
industry, to ease interface between disparate systems current and future, to accommodate 
maturing and newly developed technologies. 

Quantitative results indicated that flexibility in requirements as a program 
advantage. Primarily, requirements flexibility was used in the incremental development 
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of the TORIS system. Requirements flexibility in an EAS will allow CNSF to 
strategically acquire information systems that are capable of meeting both current and 
future needs as they evolve, prior to final contract delivery. 

Product Line Acquisition (PEA) could prove to be an effective strategy for CNSF. 
However, this approach would require detailed knowledge of core processes and 
competencies within the command. CNSF would develop IS product(s) based on these 
core processes. But to successfully execute PLA, the organization would have to first 
recognize its architecture. In other words, “procuring only an architecture in the first 
stage, procuring other core assets in the second stage, and procuring products (built from 
the core asset base) in the third stage (SEI, 2010).” Before CNSF can institute a product 
line, it must ensure that it has specific business goals, and the product line and 
architecture specifically align with those goals. The command can realize a cost-savings 
by instituting PLA in the following areas: 

• Identification and development of an architecture 

• The capability to leverage software reuse in the development of software 
interfaces and products 

• More efficient use of human capital 

• Cost estimations and metrics can be reused throughout the architecture once they 
are completed for one product 

The Acquisition Strategy Characteristic (ASC) realism is achieved by PLA’s 
development of an architecture. Developing a detailed architecture provides a framework 
to identify whether organizational goals are attainable for its architecture. If the 
organization understands its current state, it can better make strategic decisions 
concerning the integration of current legacy systems and how to best develop a software 
product that will supplant it in the future. Reuse allows the command to leverage proven 
technology and code to include in other core asset products. This method saves on code 
development and produces a software product that will integrate more seamlessly into the 
architecture, allowing for improved system stability when new capabilities and 
functionality is required in the system. Based on the statistical data discussed in Chapter 
four. Program Managers expressed that resources are of significant importance to 
successfully developing and implementing their systems. The command has placed 
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significant value on the funding resource, hence the commissioning of this three part 
study. PLA addresses this concern by allowing greater efficiency in the use of human 
resources and cost. Since systems are developed using previously proven components, 
the time and cost associated with training personnel to operate and maintain subsequent 
systems is considerably decreased, freeing up resources that can be leveraged in other 
areas of command programs. 

Performance-Based Services Acquisition (PBSA) will assist CNSF in getting the 
best quality performance from systems contractors through the use of incentives. 
However, it must meet the commands needs for performance metrics and cost-benefit 
data required to make funding decisions. There is no single acquisition strategy that will 
meet all CNSF requirements. However, certain elements of the three strategies discussed 
in this study will certainly get the command closer to the strategic decision making 
necessary to leverage its spending more efficiently and realize the most benefit in 
accomplishing mission goals. CNSF should consider their current organizational values 
and goals, technology architecture, and programs. With this information in mind, 
progress toward the development of systems that is founded upon proven systems and 
services that are already being used within the command. Aspects of the PLA hold 
several benefits for the command, specifically in meeting its concerns for long-term cost- 
savings. 

CNSF requires an acquisition strategy that will allow it to acquire quality products 
and services that have been commercially proven, from contractors that will provide high 
quality performance and support, at the best value for the command. The strategy should 
allow incremental product delivery to address technology maturation for further 
capability with standards that are open to facilitate interface with legacy and disparate 
systems. 

What are the dijferent IS Acquisition approaches ? 

There are several acquisition approaches that can be instituted to acquire systems 
and services. Aspects specific to these various approaches can be used a la carte to 
develop a tailored acquisition strategy that will meet specific organizational system 
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requirements. In Chapter II of this study, three acquisition strategies were presented and 
discussed to provide a framework for which CNSF could use for consideration, when 
strategically approaching the acquisition process necessary for developing and 
maintaining information systems meant to both enable and enhance core business 
processes. The acquisition strategies presented were: 

• Evolutionary Acquisition 

• Product Line Acquisition 

• Performance-Based Acquisition 

What types of Acquisition Approaches do other organizations with similar 
structures use ? 

This research study compared two TYCOMs with similar responsibilities and 
information systems. Commander, Naval Air Forces (CNAF) and CNSF both use 
software intensive programs to track, process, and report operational training and 
readiness in accordance with guidance and responsibilities provided by CUSFCC Fleet 
Readiness Plan. These systems: SHARP, CV SHARP, and TORIS provide an input 
training “T” pillar via DRRS-N to report the readiness status of the Navy’s Air and 
Surface Forces. The study revealed similar acquisition approaches between the two 
organizations. TORIS was spirally developed under an Evolutionary Acquisition 
Strategy (EAS) and SHARP and CV SHARP were acquired under the same approach. 
However, SHARP and CV SHARP were and are still currently being developed 
incrementally. 

Quantitative analysis also revealed that Program Managers of the studied systems 
dealt with similar obstacles in the acquisition and sustainment of their programs. The 
competition for resources was a common finding among the programs. 

The government Accountability Office (GAO) case study that was also presented, 
analyzed the acquisition approach to information systems of four different companies 
from the commercial sector. Each company presented was a leader in a different 
industry. These companies’ goals were to be the best in their industry and better leverage 
their Information Systems (IS) capability to reduce cost. While these organizations may 

not have been structured exactly as the TYCOMs, their goals are similar. The case study 
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revealed that these organizations had to significantly restructure themselves and elevate 
their priority of IS from a traditional to a more strategic approach. 

How do similar organizations measure ROI in their programs? 

The TYCOMs presented during this study maintained little to no financial data 
that could be analyzed for cost-benefit or ROI. The CV SHARP and TORIS programs 
are facing the same concerns that are looming for the financial stakeholders at CNSF, 
which is the possibility of large budget cutbacks in forthcoming fiscal years. Although 
the TORIS program has proven to add value well beyond the capabilities for which it was 
initially developed, it was not implemented for the explicit purpose of adding cost- 
savings to CNSF. The organizations presented in the case study measured ROI against 
their core mission requirements and overall business success through the use of 
information systems. 

C. RECOMMENDATIONS FOR CNSF 

1. Establish a Chief Information Officer (CIO) position 

An area of concern for CNSF is how to prioritize spending on IS to meet mission 
requirements. The organization requires a framework for which to align mission 
requirements, IS capabilities, and financial concerns. Currently, there is not enough 
knowledgeable guidance within the organization to balance these issues. A CIO, with the 
authority and support of top-level management, will assist the command in identifying 
spending priorities as they relate to the overall mission, and balance those priorities with 
current and future IS capabilities and any fiscal concerns to ensure best value and 
resource balance for the command. The CIO would ensure CNSF has a robust and 
relevant IS vision and resulting strategy, to guide the acquisition of information systems 
and capabilities. Additionally, the CIO would serve as a liaison between the technology 
centered Information Technology (IT) staff, the system contractors, and the financial 
professionals within the command to ensure all personnel involved with the acquisition of 
IS understand relevant and emergent concerns. 

The CIO can either be civilian or military. But the individual must have proven 

detailed knowledge of information systems technology, the fiscal and budgetary process, 
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and the acquisition process. If the position is to be filled using a military member, it is 
recommended that they are a senior-level officer, such as an 0-6 or 0-7. In the event that 
a civilian is chosen to fill the position, that individual should not be contracted, but 
instead have considerable experience with IS program management as a government- 
servant. 

2. Establish a Commodity Managers Advisory Board (CMAB) for IS 
Acquisitions 

In addition to the establishment of a CIO position within CNSF, it is further 
recommended that CNSF establish an advisory board of commodity manager to serve as 
guiding panel to the CIO and senior level management. This board would advise on 
issues pertaining to the acquisition of IS and capabilities to ensure the many departments 
and supporting mission areas that add to CNSF mission execution are considered. The 
CMAB should include representatives from the various core mission areas within the 
command. Program Managers from the various IS programs should be included as well. 
The purpose of this Board is to provide the CIO with relevant and emergent information 
to assist in formulating policy and guidance necessary for stakeholders to make spending, 
and functionality decisions for the command. 

3. Consideration of Implementation Steps to Progress From A 
Traditional to Strategic Approach to IS Acquisition Decision Making. 

The information in this recommendation relates to Figure 19. This section will 
provide a detailed explanation of the Figure 19 graphic and provide guidance for the 
command through each step to assist in its implementation. 

An Enterprise Architecture (EA) is the recommended Information Systems (IS) 
Strategy that will provide CNSE with the ability to ensure that its IS will evolve in 
support of its war time mission. The recommended Acquisition Strategy that will enable 
this effort is the Evolutionary Acquisition of a Software Product Eine. This strategy will 
be implemented incrementally using Performance Based Specifications. The 
Performance based specifications follow six disciplines and they are as follows : 
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• Top-Down support of strategic decision making—Proactively manage 
the organizational changes integral to the success of the initiative 

• Strategic Linkage—Provide a consistent vision throughout the 
organization, making sure the desired results reflect 
organizational strategic goals 

• Governance—Establish roles, responsibilities, and decision-making 
authorities for project implementation 

• Communications—Identify the content, medium, and frequency of 
information flow to all stakeholders 

• Risk Management—Identify, assess, monitor, and manage risks 

• Performance Monitoring—Analyze and report status—cost, schedule, and 
performance-on a regularly scheduled basis during project execution 

This recommendation evolved from several tasks that are discussed and 
depicted in Figure 20. This study identified the possible AS that could best support an 
EA. The amount of the Acquisition Strategic Characteristics that currently exists was 
examined. Third, this thesis incorporates a case study which discusses how to evolve 
from making traditional decisions to strategic decisions. The consolidated outcome from 
these tasks yielded an action plan that, implemented incrementally, could provide a 
roadmap for CNSF to ensure its IS evolve in step with its operational mission. The 
action plan is based upon five steps. 

CNSF should consider restructuring their current organization in order to facilitate 
strategic IS decision making. Senior-level commitment is necessary and will set 
mandates and ensure continued support. They will assist in developing the IS Vision, 
proper leveraging of IS funding, and serve as a subject-matter-expert for technology 
requirements. Also, a Commodity Managers Advisory Board (CMAB) would need to be 
established to serve as an approval panel for technology request that not only impact the 
operational budget, but also the infrastructure. This group is comprised of diverse 
experts within the organization that understand either the operational or technical 
requirements along with their impact on the organization. Changes to the overall 
leadership structure will enable the organization to better manage their requirements. 

CNSF will need to conduct and internal analysis of its spending, mission 
requirements, and current system and program performance. A spend analysis can 
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provide insight on what items are being purchased, who is being paid, who ordered it, and 
how much. A performance analysis will examine whether the program is still needed or 
should it be modified to meet user needs. The performance of the contractor should also 
be assessed to ensure the command is receiving the highest quality outcome and 
contractor performance is meeting criteria set-forth in the contract statement of work 
(SOW). The ability of the ROI’s to meet the user’s requirements should also be 
analyzed. If no ROI’s exist, then program performance tasks must be determined. 
Analyze these tasks to discern what performance levels are acceptable. As these tasks are 
complete, feedback is sent to the leadership to ensure their approval and to increase 
awareness of the organization’s posture. Conducting an internal analysis will increase 
CNSF’s ability to make informed investment decisions. 

The organization should conduct assessments on how well current program 
functions are being performed. The commodity managers will understand the current 
program and will make assessments as to how it can be done better. They will conduct a 
study and analysis of alternative systems and capabilities that may provide more 
efficiency, and require fewer resources. A risk management mitigation assessment will 
develop an alternative plan for sustaining the organization when a key resource is lost. 
This plan will serve to minimize the impact of the lost. The contractor assessment will 
provide feedback on whether a contractor is performing the tasks they were hired to do. 
As these tasks are complete, feedback is sent back to the leadership for approval and 
authorization to proceed with needed action. Conducting assessments will offer CNSF 
the option of capitalizing on new technology that has been researched and vetted by a 
team of experts. 

From the previous step, CNSF will have sound information that will guide them 
in executing needed tasks. Once the role of IS has been deemed a combat multiplier, 
CNSF will need to establish an IS funding line for the POM cycle. This will ensure that 
the IT program will have the funds to evolve in step with the combat systems. Each 
program needs to have a baseline with version control to support planned growth. Next 
CNSF will be able to balance spending over requirements with the help of the 
Commodity Managers Advisory Board. Finally, the command will need to assign 
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accountability to ensure tasks have oversight. As these tasks are complete, feedback is 
sent to the leadership for approval and endorsement of actions taken. By executing these 
tasks, CNSF’s ability to manage mission changes will be enhanced because it not only 
meets a user needs, but will also support the organization’s strategic goals. 

Finally, CNSF will need to manage the resources of time, money, people, and 
facilities against the requirements of cost, schedule, and performance. Feedback to the 
leadership will indicate the organization’s ability to be successful or the need to 
realloeate resources. By managing the resources and requirements, CNSF can optimize 
their decision-making ability. 
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4. Opportunities for Future Research 

Decision making for the acquisition of IS within CNSF requires greater 
knowledge of what capabilities will provide the greatest return on investment (ROI) for 
the command. Additionally, the command must identify the most effective method for 
either cutting or increasing program spending. Therefore, this research should result in a 
spending analysis, a cost-benefit analysis, and an ROI analysis of CNSF programs to 
provide better knowledge to the command so that it can make more informed decisions 
on IS spending. 

To execute the analyses recommended by this research we recommend that CNSF 
utilize both student research and resources within its own command to develop analysis 
and recommendations to enhance the commands decision-making process so that it may 
leverage it in future IS acquisition decisions. 
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APPENDIX A 


Survey Responses- Realism Questions 


Custom Data 

What significance 
has the 

implementation of 
tills program had 
on the Command's 
overall mission? 

How significant 
has Command 
support been to 
the overall 
success of this 
system? 

How significant 
did force-fit 
measures, that 
can't be 
compromised, 
affect the 
development or 
operation of 
this program? 

How important 
have the role of 
priority levels, 
when 

compared to 
similar 

programs, 
affected 
decisions made 
for your 
particular 
program? 

To what 
extent were 
realistic 
expectations 
important 
with regard to 
this program's 
scheduling 
and resource 
requirements? 


Response 

Response 

Response 

Response 

Response 

1002 

1 

1 

1 

1 

1 

1012 

1 

1 

2 

2 

1 

cnsf 

2 

3 

4 

4 

3 

1007 

1 

2 

3 

3 

2 

1006 

1 

1 

4 

2 

2 

1009 

1 

2 

3 

2 

2 

1010 

1 

2 

5 

4 

4 

1001 

1 

1 

4 

3 

2 

3 

4 

1 

3 

1 

2 

4 

1 

1 

2 

3 

1 

5 

2 

1 

1 

2 

1 
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Survey Responses- Stability Questions 


Custom 

Data 

How significant 
will changes by 
the user or in 
capability cause 
a major 

disrupHon of the 
technical 
progress needed 
for the system? 

How significant 
will the changes 
In policy or 
leadership affect 
the direction of 
this project? 

How significant 
is the impact of 
having higher 
level supporters 
speaWng on 
behalf of your 
program? 

How significant 
could the use 
of an IPT 
improve the 
success of this 
program? 

How would 
changes in 
industry, loss 
of a major 
contractor, or 
failure to 
modernize 
impact this 
program? 

How would 
changes in 
personnel that 
cause a lack of 
accountability or 
a loss of audit 
trail affect this 
program? 

How important 
will It be to 
operate with 
objecBves, 
approaches, 
and control 
procedures to 
ensure progress 
in the program? 


Response 

Response 

Response 

Response 

Response 

Response 

Response 

1002 

1 

1 

1 

1 

1 

1 

1 

1012 

4 

2 

2 

5 

2 

1 

1 

cnsf 

4 

3 

1 

2 

1 

2 

2 

1007 

4 

2 

1 

3 

1 

3 

2 

1006 

4 

3 

1 

3 

1 

1 

5 

1009 

3 

1 

1 

2 

1 

1 

1 

1010 

5 

4 

2 

4 

2 

2 

3 

1001 

4 

2 

1 

3 

1 

2 

2 

3 

2 

1 

1 

3 

1 

1 

1 

4 

2 

3 

2 

3 

1 

2 

2 

5 

3 

1 

1 

4 

2 

2 

3 
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Survey Responses- Resource Balance 
Questions 


Custom 

Data 

How significant 
have the 
management 
of budget for 
the personnel, 
facilities, and 
time affected 
the successful 
development or 
operation of 
this project? 

How important 
was budgeting 
from inception to 
the overall 
execution of this 
program? 

How important 
would being part 
of a continuous 
funding vehicle 
affect the 
financial stability 
of this project? 





1002 

1 

1 

1 

1012 

1 

2 

2 

cnsf 

2 

2 

2 

1007 

2 

2 

2 

1006 

2 

1 

1 

1009 

2 

2 

1 

1010 

2 

2 

2 

1001 

2 

2 

2 

3 

1 

1 

1 

4 

1 

3 

3 

5 

1 

1 

1 
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Survey Responses - Flexibility Questions 


Custom 

Data 

How significant 
would it be if the 
user/customer 
was readily 
available to 

answer 

questions? 

How significant 
would it be if 
the contract for 
the program 
had built in 
flexibility 
allowing 
changes to 
happen without 
consultation 
with the 
customer? 

How important 
has risk 
management 
planning been to 
maintaining the 
program 
schedule? 

How much of an 
impact to the 
design was 
attributed to 
ensuring that the 
program could 
support a balance 
among 
performance, 
productivity, and 
logistics? 


Response 


Response 

Response 

1002 

1 

1 

1 

1 

1012 

3 

2 

1 

2 

cnsf 

2 

2 

4 

4 

1007 

3 

2 

2 

3 

1006 

1 

4 

3 

3 

1009 

1 

2 

1 

2 

1010 

3 

4 

3 

2 

1001 

2 

3 

2 

2 

3 

1 

1 

2 

1 

4 

1 

2 

3 

2 

5 

3 

4 

2 

3 
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Survey Responses- Risk Management 
Questions 


Custom 

Data 

How significant 
have 

uncontroliabie 
requirement 
shifts affected 
this program 

How significant 
have the roie 
of natural 
disasters 
affected the 
development or 
operation of 
this program? 

What 

significance has 
funding changes 
had on the 
direction of this 
program? 

How significant 
have 

government 
standards 
affected the 
development or 
operation of this 
program? 

How important 
have changes 
with the 
contractor 
affected the 
development or 
operation of this 
program? 







1002 

1 

1 

3 

3 

1 

1012 

1 

5 

3 

2 

1 

cnsf 

3 

4 

2 

2 

4 

1007 

4 

4 

3 

1 

2 

1006 

5 

5 

1 

1 

5 

1009 

2 

4 

2 

2 

1 

1010 

3 

5 

4 

4 

1 

1001 

4 

5 

3 

3 

2 

3 

1 

5 

1 

1 

1 

4 

2 

4 

4 

3 

3 

5 

2 

5 

2 

1 

2 
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Survey Responses- Risk Management 
Questions 












































Survey Responses - Time and Difficulty 
Questions 


Custom Dc 

How iong did it 
take to 

impiement this 
program? 

Current 

baseline 

integration: 


Program 

Certification: 


What percentage 
of success or 
failure would you 
rate this 
program's 
development or 
operation? 


Open-Ended 

Response 

How long did it 
take? 

On a scale or 1- 
7, with 7 being 
most difficuit, 
how would you 
rate the 
difficulty? 

How long did it 
take? 

On a scale or 1- 
7, with 7 being 
most difficuit, 
how would you 
rate the 
difficulty? 

Open-Ended 

Response 

1002 

2 Years 

1 year 

4 

2 years 

5 

85% Sucess 

1012 

Five years, and 
stiil deveioping 
new usage 
features 

18 months 
(further detaiis 
in email) 

7 

2 years 

7 

80% success 

cnsf 

which time? the 
iatest version 
took severai 
months of 
testing to feei 
comfortabie in 
going forward 
with the 
upgrade from 
SharePoint 2003 
to SharePoint 
2007 

see #1. 

5 

not sure. LANT 
did the 
certification 
since the 
hardware is 
housed there. 

6 

75 

n 

2 years for initiai 
impiementation - 
10 year program 
totai 

1 year 

5 

10 months 

6 

90% successful 

1006 

n/a 

6mth, to do 
80% of Navy; 
base pr ocess 
and approval 

process 

5 

1 yr, ATO was 
initaiiy painfui, 
once approved 
iearned how to 
manage it and 
became easier 
due to having 
dedicated 
people, a 
process, and a 
reoccurring 
window to keep 
everyone 
focused 

3 

100% 


99 







































Survey Responses - Time and Difficulty 
Questions 


Custom Dc 

How long did it 
take to 

implement this 
program? 

Current 

baseline 

integration: 


Program 

Certification: 


What percentage 
of success or 
failure would you 
rate this 
program's 
development or 
operation? 

1009 

Continiuos 

implementation 

7 Weeks with a 
few outlying 
ships 

5 

Unkown 

4 

95% 

1010 

1-2 years 

1 year 

6 

1 year 

5 

100 

1001 

About 18 
months to 
introduce TFOM 
into TORIS and 
deploy to the 
Fleet 

8 months 

4 

18 months 

6 

85% in providing 
a reliable, web- 
based data 
engine to 
support 

SURFACE FORCE 
training and 
readiness 
process 
management 

3 






100 

4 

Don't know, 
TORIS has been 
implemented 
since I've been 
here in 2007. 
Upgrades to 
system are 
generally 
delayed to fleet. 

TBD 

6 changes and 
new demands 
from users and 
leadership 
around the globe 
continue to slow 
process of 
integration. 

unknown (I'm an 
operator, not a 
developer) 

unknown 

90. Distance 
and non¬ 
availability of 
users make 
problems difficult 
to solve 

5 

Depends - 6 
months to 6 

years 

5 years 

5 

1 year 

4 

100% success 


100 




































Survey Responses - Time and Difficulty 
Questions 




















SutA/ey Responses - Time and Difficulty 
Questions 


Custom Dc 

On a scale or 1- 
7, with 7 being 
most difficult, 
how would you 
rate this 
program's 
piocurement? 

Was this 
program 
previously used 
by other 
commands? 

1009 

5 

No 

1010 

5 

no 

1001 

5 

no 

3 


YES 

4 

unknown 
(operator only) 

at least used by 
Navy Afloat 
Training 
Commands PAC 
and LANT 

5 

6 - very difficult 
to procure 

no 




















APPENDIX B 


Data for Figure 11 


mean 
Std error 


CNSFWeb TORIS 

2 2 

3 1 

5 3 

4 2 

4 2 

5 3 

4 2 

2 1 

3 3 

2 1 

2 1 

3 2 

2 1 

2 2 

2 1 

3 1 

3 2 

4 2 

3 2 

3 2 

5 4 

3 2 

3 2 

3 2 

2 1 

2 1 

2 1 

CNSF Web TORIS 

3 1.814815 

0.19245 0.151319 


SHARP 

1 

2 

3 

3 
2 

4 
2 
1 
3 
1 
3 
2 
2 
2 
2 
3 
2 
2 

3 

4 
4 
3 
1 
2 
1 
2 
2 

SHARP 

2.296296 

0.175532 


CV SHARP 
1 
1 
2 
2 
1 

4 
2 
2 

5 
2 
1 
1 
1 
2 
2 
3 
2 
1 
2 
1 
5 
3 
2 
1 
2 
1 
1 

CV SHARP 
1.962963 
0.22317 
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Data for Figure 11 


CNSFWeb 


TORIS 


SHARP 


CVSHARP 


Mean 

3 

Mean 

1.814815 

Mean 

2.296296 

Mean 

1.962963 

Standard Error 

0.19245009 

Standard E 

0.151319 

Standard E 

0.175532 

Standard Ei 

0.22317 

Median 

3 

Median 

2 

Median 

2 

Median 

2 

Mode 

2 

Mode 

2 

Mode 

2 

Mode 

1 

Standard Deviation 

1 

Standard D 

0.786278 

Standard D 

0.91209 

Standard D 

1.159625 

Sample Variance 

1 

Sample Vat 

0.618234 

Sample Vat 

0.831909 

Sample Vat 

1.344729 

Kurtosis 

-0.37538462 

Kurtosis 

0.822322 

Kurtosis 

-0.49482 

Kurtosis 

1.956855 

Skewness 

0.74769231 

Skewness 

0.863193 

Skewness 

0.332369 

Skewness 

1.515321 

Range 

3 

Range 

3 

Range 

3 

Range 

4 

Minimum 

2 

Minimum 

1 

Minimum 

1 

Minimum 

1 

Maximum 

5 

Maximum 

4 

Maximum 

4 

Maximum 

5 

Sum 

81 

Sum 

49 

Sum 

62 

Sum 

53 

Count 

27 

Count 

27 

Count 

27 

Count 

27 


Results from ASC Questions 


3.5 



CNSF Web TORIS SHARP CV SHARP 

Programs Reviewed 
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Data - Figure 12 


Time 


CNSFWeb TORIS 

SHARP 

CVSHARP 

CNSFWEB 

2 

Implement 2 

2 

2 ! 


1 

Integrate 1 

2 

1 


1 

Certificatio 1 

1 

1 


TORIS 2 

2 
1 

SHARP 2 

1 
1 

CVSHARP 5 

2 
2 


By Program Comparison 



■ Implement 

■ Integrate 

■ Certification 


Programs 
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Data - Figure 13 


Likert Scale 


CNSFWEB 

6 

CNSFWeb TORIS 

SHARP 

CVSHARP 


5 

Integration 6 

5 

5 7 


6 

Procureme 5 

5 

3 7 

TORIS 

5 

Certificatio 6 

5 

6 7 


5 

5 

SHARP 5 

3 

6 

CVSHARP 7 

7 
7 


Level of Difficulty 



■ Integration 

■ Procurement 

■ Certification 


CNSFWeb TORIS SHARP CVSHARP 


Programs 
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Data - Figure 14 


Bin Frequency 

1 5 

2 12 

3 7 

4 3 

5 0 

More 0 


SHARP 

s 

c 

9) 

3 

? 


Question Response Distribution 


Bin Frequency 
1 10 

2 13 

3 3 

4 1 

5 0 

More 0 
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Data - Figure 14 


Bin 


Frequency 


1 

11 


2 

11 


3 

2 


4 

1 


5 

2 

More 


0 






CV SHARP 


15 n 



V 

10 n 



O' 

w 

Ik 

5 ' 





1 

2 3 4 5 More 




Question Response Distribution 


Bin 


Frequency 


1 

0 


2 

10 


3 

10 


4 

4 


5 

3 

More 


0 



CNSF Web 


15 n 



1 2 3 4 5 More 

Question Response Distribution 
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Data- Table 5 



CNSF Web TORIS 

SHARP 

CV SHARP 


Realism 

2 

2 

1 

1 


3 

1 

2 

1 


5 

3 

3 

2 


4 

2 

3 

2 


4 

2 

2 

1 


CNSF Web TORIS 

SHARP 

CV SHARP 


Stabilty 

5 

3 

4 

4 


4 

2 

2 

2 


2 

1 

1 

2 


3 

3 

3 

5 


2 

1 

1 

2 


2 

1 

3 

1 


3 

2 

2 

1 


CNSF Web TORIS 

SHARP 

CV SHARP 


Resource B 

2 

1 

2 

1 


2 

2 

2 

2 


2 

1 

2 

2 


CNSF Web TORIS 

SHARP 

CV SHARP 


Flexibilty 

3 

1 

3 

3 


3 

2 

2 

2 


4 

2 

2 

1 


3 

2 

3 

2 


CNSF Web TORIS 

SHARP 

CV SHARP 


Risk Mngt 

3 

2 

4 

1 


5 

4 

4 

5 


3 

2 

3 

3 


3 

2 

1 

2 


3 

2 

2 

1 


2 

1 

1 

2 


2 

1 

2 

1 


2 

1 

2 

1 


Table 5 

Mean (Std, Dev) 




Variable 

CNSF Web 

TORIS 

SHARP 

CV SHARP 

Realism 

3.6(1.1) 

2(.7) 

2.2(.8) 

1.4(.5) 

Stability 

3(1.2) 

1.9(.9) 

2.3(1.1) 

2.4(1.5) 

Resource Balance 

2(0) 

1.3(.6) 

2(0) 

1.7(.6) 

Flexibility 

3.3(.5) 

1.8(.5) 

2.5(.6) 

2(.8) 

Risk Management 

2.9(1) 

1.9(1) 

2.4(1.2) 

2(1.4) 


Variable 

CNSF Web 

TORIS 

SHARP 

CV SHARP 

Realism 

3.6(1.1) 

2(.7) 

2.2(.8) 

1.4(.5) 

Stability 

3(1.2) 

1.9(.9) 

2.3(1.1) 

2.4(1.5) 

Resource Balance 

2(0) 

1.3(.6) 

2(0) 

1.7{.6) 

Flexibility 

3.3(.5) 

1.8(.5) 

2.5(.6) 

2(.8) 

Risk Management 

2.9(1) 

1.9(1) 

2.4(1.2) 

2(1.4) 
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Realism 


CNSFWeb 


TORIS 


SHARP 


CVSHARP 


Mean 

3.6 

Mean 

2 

Mean 

2.2 

Mean 

1.4 

Standard Error 

0.509902 

Standard E 

0.316228 

Standard E 

0.374166 

Standard Ei 

0.244949 

Median 

4 

Median 

2 

Median 

2 

Median 

1 

Mode 

4 

Mode 

2 

Mode 

2 

Mode 

1 

Standard Deviation 

1.140175 

Standard D 

0.707107 

Standard D 

0.83666 

Standard D 

0.547723 

Sample Variance 

1.3 

Sample Vai 

0.5 

Sample Var 

0.7 

Sample Var 

0.3 

Kurtosis 

-0.17751 

Kurtosis 

2 

Kurtosis 

-0.612245 

Kurtosis 

-3.333333 

Skewness 

-0.4048 

Skewness 

0 

Skewness 

-0.512241 

Skewness 

0.608581 

Range 

3 

Range 

2 

Range 

2 

Range 

1 

Minimum 

2 

Minimum 

1 

Minimum 

1 

Minimum 

1 

Maximum 

5 

Maximum 

3 

Maximum 

3 

Maximum 

2 

Sum 

18 

Sum 

10 

Sum 

11 

Sum 

7 

Count 

5 

Count 

5 

Count 

5 

Count 

5 

Stability 

CNSFWeb 


TORIS 


SHARP 


CV SHARP 


Mean 

3 

Mean 

1.857143 

Mean 

2.285714 

Mean 

2.428571 

Standard Error 

0.436436 

Standard E 

0.340068 

Standard E 

0.42056 

Standard Ei 

0.571429 

Median 

3 

Median 

2 

Median 

2 

Median 

2 

Mode 

2 

Mode 

1 

Mode 

2 

Mode 

2 

Standard Deviation 

1.154701 

Standard D 

0.899735 

Standard D 

1.112697 

Standard D 

1.511858 

Sample Variance 

1.333333 

Sample Vai 

0.809524 

Sample Var 

1.238095 

Sample Var 

2.285714 

Kurtosis 

-0.15 

Kurtosis 

-1.816609 

Kurtosis 

-0.944379 

Kurtosis 

-0.196875 

Skewness 

0.909327 

Skewness 

0.353045 

Skewness 

0.248875 

Skewness 

1.000425 

Range 

3 

Range 

2 

Range 

3 

Range 

4 

Minimum 

2 

Minimum 

1 

Minimum 

1 

Minimum 

1 

Maximum 

5 

Maximum 

3 

Maximum 

4 

Maximum 

5 

Sum 

21 

Sum 

13 

Sum 

16 

Sum 

17 

Count 

7 

Count 

7 

Count 

7 

Count 

7 

Resource Balance 

CNSFWeb 


TORIS 


SHARP 


CV SHARP 


Mean 

2 

Mean 

1.333333 

Mean 

2 

Mean 

1.666667 

Standard Error 

0 

Standard E 

0.333333 

Standard E 

0 

Standard Ei 

0.333333 

Median 

2 

Median 

1 

Median 

2 

Median 

2 

Mode 

2 

Mode 

1 

Mode 

2 

Mode 

2 

Standard Deviation 

0 

Standard D 

0.57735 

Standard D 

0 

Standard D 

0.57735 

Sample Variance 

0 

Sample Vai 

0.333333 

Sample Var 

0 

Sample Var 

0.333333 

Kurtosis 

#DIV/0! 

Kurtosis 

#DIV/0l 

Kurtosis 

#DIV/0! 

Kurtosis 

#DIV/0l 

Skewness 

#DIV/0! 

Skewness 

1.732051 

Skewness 

#DIV/0! 

Skewness 

-1.732051 

Range 

0 

Range 

1 

Range 

0 

Range 

1 

Minimum 

2 

Minimum 

1 

Minimum 

2 

Minimum 

1 

Maximum 

2 

Maximum 

2 

Maximum 

2 

Maximum 

2 

Sum 

6 

Sum 

4 

Sum 

6 

Sum 

5 

Count 

3 

Count 

3 

Count 

3 

Count 

3 
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Data- Table 5 


Flexibility _ 

CNSF Web TORIS SHARP CV SHARP 


Mean 

3.25 Mean 

Standard E 

0.25 Standard Error 

Median 

3 Median 

Mode 

3 Mode 

Standard D 

0.5 Standard Deviation 

Sample Var 

0.25 Sample Variance 

Kurtosis 

4 Kurtosis 

Skewness 

2 Skewness 

Range 

1 Range 

Minimum 

3 Minimum 

Maximum 

4 Maximum 

Sum 

13 Sum 

Count 

4 Count 


Risk Management _ 

CNSF Web TORIS 


1.75 Mean 2.5 Mean 2 

0.25 Standard Errc 0.28867513 Standard Errc 0.40824829 
2 Median 2.5 Median 2 

2 Mode 3 Mode 2 

0.5 Standard Dev 0.57735027 Standard Dev 0.81649658 
0.25 Sample Variai 0.33333333 Sample Variai 0.66666667 
4 Kurtosis -6 Kurtosis 1.5 

-2 Skewness 0 Skewness 0 

1 Range 1 Range 2 

1 Minimum 2 Minimum 1 

2 Maximum 3 Maximum 3 

7 Sum 10 Sum 8 

4 Count 4 Count 4 


SHARP CVSHARP 


Mean 

Standard E 

Median 

Mode 

Standard D 

Sample Var 

Kurtosis 

Skewness 

Range 

Minimum 

Maximum 

Sum 

Count 


2.875 Mean 

0.350382 Standard Error 
3 Median 
3 Mode 

0.991031 Standard Deviation 
0.982143 Sample Variance 
2.973091 Kurtosis 
1.486055 Skewness 
3 Range 
2 Minimum 
5 Maximum 
23 Sum 
8 Count 


1.875 Mean 

0.35038244 Standard Errc 
2 Median 

2 Mode 

0.99103121 Standard Dev 
0.98214286 Sample Variai 
2.97309091 Kurtosis 
1.48605539 Skewness 

3 Range 

1 Minimum 

4 Maximum 
15 Sum 

8 Count 


2.375 Mean 

0.41992771 Standard Errc 
2 Median 

2 Mode 

1.18773494 Standard Dev 
1.41071429 Sample Variai 
-1.2292902 Kurtosis 
0.39432572 Skewness 

3 Range 

1 Minimum 

4 Maximum 
19 Sum 

8 Count 


2 

0.5 

1.5 

1 

1.41421356 

2 

2.47142857 

1.61624407 

4 
1 

5 

16 

8 
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CNSFWeb Rank-web TORIS Rank-TORi: SHARP Rank-SHAFCV SHARP Rank-CVSHARP 


Realism 

2 

16.5 

2 

16.6 

1 

6.5 

1 

6.5 


3 

27 

1 

6.5 

2 

16.5 

1 

6.5 


5 

46 

3 

37 

3 

37 

2 

16.5 


4 

34.5 

2 

16.5 

3 

27 

2 

16.5 


4 

34.5 

2 

16.5 

2 

16.5 

1 

6.5 

Tl = 


158.5 T2 = 


93.1 T3 = 


103.5 T4 = 


52.5 

Stabilty 

5 

5.5 

3 

3.5 

4 

4.5 

4 

4.5 


4 

4.5 

2 

2.5 

2 

2.5 

2 

2.5 


2 

2.5 

1 

1.5 

1 

1.5 

2 

2.5 


3 

3.5 

3 

3.5 

3 

3.5 

5 

5.5 


2 

2.5 

1 

1.5 

1 

1.5 

2 

2.5 


2 

2.5 

1 

1.5 

3 

3.5 

1 

1.5 


3 

3.5 

2 

2.5 

2 

2.5 

1 

1.5 

Tl = 


24.5 T2 = 


16.5 T3 = 


19.5 T4 = 


20.5 

Resource B 

2 

2.5 

1 

1.5 

2 

2.5 

1 

1.5 


2 

2.5 

2 

2.5 

2 

2.5 

2 

2.5 


2 

2.5 

1 

1.5 

2 

2.5 

2 

2.5 

Tl = 


7.5 T2 = 


5.5 T3 = 


7.5 T4 = 


6.5 

Flexibilty 

3 

3.5 

1 

1.5 

3 

3.5 

3 

3.5 


3 

3.5 

2 

2.5 

2 

2.5 

2 

2.5 


4 

4 

2 

2.5 

2 

2.5 

1 

1.5 


3 

3.5 

2 

2.5 

3 

3.5 

2 

2.5 

Tl = 


14.5 T2 = 


9 T3= 


12 T4 = 


10 

Risk Mngt 

3 

3.5 

2 

2.5 

4 

4.5 

1 

1.5 


5 

5.5 

4 

4.5 

4 

4.5 

5 

5.5 


3 

3.5 

2 

2.5 

3 

3.5 

3 

3.5 


3 

3.5 

2 

2.5 

1 

1.5 

2 

2.5 


3 

3.5 

2 

2.5 

2 

2.5 

1 

1.5 


2 

2.5 

1 

1.5 

1 

1.5 

2 

2.5 


2 

2.5 

1 

1.5 

2 

2.5 

1 

1.5 


2 

2.5 

1 

1.5 

2 

2.5 

1 

1.5 

Tl = 


27 T2 = 


19 T3 = 


23 T4 = 


20 
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nMaJiguilji 


Risk 


Realism 


CNSF Web TORIS 


SHARP 

CV SHARP 

Kruskal-Wallis Test 

1 

16.5 

16.6 

6.5 

6.5 



1 

27 

6.5 

16.5 

6.5 

Group 

Rank Sum 

Observatior 

46 

37 

37 

16.5 

CNSF Web 

76.5 

5 

34.5 

16.5 

27 

16.5 

TORIS 

52.5 

5 

34.5 

16.5 

16.5 

6.5 

SHARP 

54 

5 

158.5 

93.1 

103.5 

52.5 

CVSHARP 

27 

5 





H Stat 


7.02 





df 


3 





p-value 


0.0713 





cbi-squared Critical 

7.8147 


Resources CNSF 

TORIS 


SHARP 

CV SHARP 


Kruskal-Wallis Test 

1 


2.5 

1.5 

2.5 

1.5 




1 


2.5 

2.5 

2.5 

2.5 


Group 

Rank Sum 

Observatior 


2.5 

1.5 

2.5 

2.5 


CNSF 

24 

3 







TORIS 

12 

3 







SHARP 

24 

3 







CVSHARP 

18 

3 







H Stat 


2.5385 







df 


3 







p-value 


0.4684 







chi-squared Critical 

7.8147 

CNSF TORIS 

SHARP 


CV Sharp 


Kruskal-Wallis Test 



3.5 

2.5 

4.5 

1.5 






5.5 

4.5 

4.5 

5.5 


Group 

Rank Sum 

Observations 

3.5 

2.5 

3.5 

3.5 


CNSF 

176 

8 


3.5 

2.5 

1.5 

2.5 


TORIS 

106 

8 


3.5 

2.5 

2.5 

1.5 


SHARP 

139 

8 


2.5 

1.5 

1.5 

2.5 


CV Sharp 

107 

8 


2.5 

1.5 

2.5 

1.5 






2.5 

1.5 

2.5 

1.5 


HStat 


4.6676 







df 


3 







p-value 


0.1978 







chi-squared Critical 

7.8147 
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Stabllty 


TORIS 


SHARP 

CV SHARP 

Kruskal-WallisTest 


5.5 

3.5 

4.5 

4.5 




4.5 

2.5 

2.5 

2.5 

Group 

Rank Sum 

Observatio 

2.5 

1.5 

1.5 

2.5 

CNSF 

131 

7 

3.5 

3.5 

3.5 

5.5 

TORIS 

78 

7 

2.5 

1.5 

1.5 

2.5 

SHARP 

99 

7 

2,5 

1.5 

3.5 

1.5 

CVSHARP 

98 

7 

3.5 

2.5 

2.5 

1.5 








HStat 


3.0422 





df 


3 





p-value 


0.3852 





chi-squared 

Critical 

7.8147 


Flexibility CNSF 


TORIS 


SHARP 

CV SHARP 

3.5 

1.5 

3.5 

3.5 

3.5 

2.5 

2.5 

2.5 

4 

2.5 

2.5 

1.5 

3.5 

2.5 

3.5 

2.5 


Kruskal-Wallis Test 


Group 

Rank Sum 0 

bservatio 

CNSF 

53.5 

4 

TORIS 

19.5 

4 

SHARP 

37 

4 

CV SHARP 

26 

4 

H Stat 


7.318 

df 


3 

p-value 


0.0624 

chi-squared Critical 

7.8147 
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Realism 


CNSFWEB 2 

3 
5 

4 
4 

TORIS 2 

1 
3 
2 
2 

SHARP 1 

2 
3 
3 
2 

CVSHARP 1 
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2 
1 


Realism 

CNSFWEB TORiS 
CNSFWEB SHARP 
CNSFWEB CVSHARP 
TORIS SHARP 

TORIS CVSHARP 
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TORIS 

SHARP 
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1 

1 

2 
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2 

3 

2 

2 


TORIS 
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2 

1 

1 

1 

3 

2 

2 

2 

2 
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SHARP 
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2 

1 

3 

2 

3 

2 

2 

1 


CNSFWeb 

TORIS 
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2 
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1 
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CNSFWeb 

SHARP 

2 

1 

3 

2 

5 

3 

4 

3 

4 

2 


CNSFWeb 

CVSHARP 
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1 
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Data - Figure 16 


Wilcoxon Rank Sum Test 


Wilcoxon Rank Sum Test 



Rank Sum Observations 


Rank Sum Observations 

CNSFWeb 

37 5 

CNSFWEB 

36 5 

TORIS 

18 5 

SHARP 

19 5 

2 Stat 

1.9845 

2 Stat 

1.7756 

P(Z<=z) one-tail 

0.0236 

P(Z<= 2 ) one-tail 

0.0379 

2 Critical one-tail 

1.2816 

2 Critical one-tail 

1.2816 

|P(Z<=z) two-tail 

0.0472] 

|P(Z<= 2 ) two-tail 

0.0758] 

2 Critical two-tail 

1.6449 

2 Critical two-tail 

1.6449 


Wilcoxon Rank Sum Test 

Wilcoxon Rank Sum Test 



Rank Sum Observations 


Rank Sum 

Observations 

TORIS 

33.5 5 

SHARP 

34.5 

5 

CVSHARP 

21.5 5 

CVSHARP 

20.5 

5 

2 Stat 

1.2534 

2 Stat 

1.4623 


P(Z<=z) one-tail 

0.105 

P(Z<= 2 ) one-tail 

0.0718 


2 Critical one-tail 

1.2816 

2 Critical one-tail 

1.2816 


P(Z<=z) two-tail 

0.21 

P(Z<=z) two-tail 

0.1436 


2 Critical two-tail 

1.6449 
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Data - Figure 17 
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Realism Data - Table 6 


CNSFWEB 


TORIS 


SHARP 


CVSHARP 


CNSFWEB 


TORIS 


SHARP 


CVSHARP 


Realism Realism 


2 

3 
5 

4 
4 
2 
1 
3 
2 
2 
1 
2 
3 
3 
2 
1 
1 
2 
2 
1 


2 

3 
5 

4 
4 
2 
1 
3 
2 
2 
1 
2 
3 
3 
2 
1 
1 
2 
2 
1 


Stability Realism 


5 

4 

2 

3 

2 

2 

3 

3 

2 

1 

3 
1 
1 
2 

4 
2 
1 
3 
1 

3 
2 

4 
2 
2 


2 

3 
5 

4 
4 
2 
1 
3 
2 
2 
1 
2 
3 
3 
2 
1 
1 
2 
2 
1 


Spearman Rank Correlation 


Realism and Realism 

Spearman Rank Correlation 1 

z Stat 4.3589 

P(Z<=z) one tail 0 

z Critical one tail 1.2816 

P(Z<=z) two tail 0 

z Critical two tail 1.6449 


Spearman Rank Correlation 

Stability and Realism 
Spearman Rank Correlation 
z Stat 

P(Z<=z) one tail 
z Critical one tail 
P(Z<=z) two tail 
z Critical two tail 


-0.0069 

-0.0302 

0.488 

1.2816 

0.976 

1.6449 


119 



Realism Data - Table 6 
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Realism Data - Table 6 
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Realism Data - Table 6 
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stability Data - Table 6 
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Flexibility Data - Table 6 
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Resource Balance Data- Table 6 
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Resource Balance Data- Table 6 
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Resource Balance Data- Table 6 
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Risk Management Data - Table 6 
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Risk Manasement Data - Table 6 
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Risk Management Data - Table 6 
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APPENDIX C 


The information within this appendix is original analysis of the GAO’s case study 
of six commercial organizations that sought to move from a traditional to strategic 
approach on IS acquisition decision making. The analysis reviews the four concepts 
recommended by the GAO and discusses the lessons learned from the study. The 
analysis was used to provide a background of recommendations to be provided to 
Commander, Naval Surface Forces in its transformation toward strategic IS acquisition 
decision making. 

GAO Case Study 

Title: 

Adopting a Strategic Approach for service acquisitions will leverage your 
resources and reduce your operating costs 

Abstract: 

The government Accountability Office (GAO) conducted a study of six 
companies from the commercial industry in order to capture lessons learned on how they 
evolved from traditional to strategic service acquisitions. Strategic decision making was 
based upon four concepts: 

• Commitment 

• Knowledge 

• Change 

• Support 

Any change effort must have top-level leader support. Commitment to any change 
or initiative should have upper-management support and be pushed down through the 
various echelons of the organization. The organization must have detailed knowledge of 
their core values and processes in order to facilitate effective decision making. A 
company should perform a detailed spend analysis in order to gain necessary knowledge 
and become proactive in decision making vice reactive. Third, an organization must 
create the structure, processes, and roles for strategic decision making. To meet the goals 
of this principle, an organization must look to experts in the commodities they are hoping 
to procure. Ultimately, for these changes to remain in place there must be sustained 
leader support, open communication channels, and credible metrics for process utilization 
and compliance. By embracing these four concepts, the organizations in this study were 
able to better leverage their resources, reduce their operating costs, better manage their 
service providers, and gain better quality from services rendered. 

Background: 

This case study shows how companies in the commercial industry improved their 
business processes and saved money by adopting a strategic approach for service 
acquisitions. The GAO conducted this study as a result of growing concern about federal 
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agency spending. The GAO studied six companies to determine what lessons learned 
could be drawn from their experiences. This study demonstrates how to overcome 
traditional business practices and reap the benefits of strategic decision making. 

Hypothesis: 

A Strategic or Enterprise approach to service acquisitions could better leverage 
resource dollars and reduce operational costs. 

Inquiry Questions that support the hypothesis: 

• Why must you secure Up-front Commitment from top leaders? 

• What are the benefits for obtaining better knowledge on service spending? 

• How can an organization improve its service acquisition decisions, which 
are applicable across the organization, by creating enterprise structure, 
processes, and supporting roles? 

• How can an organization improve service acquisition decision making 
through sustained leaderships, communication, and metrics? 

Methodology/Results: 

Overview of the GAO Study 

The companies in this study realized that they did not have control of money 
flows when acquiring services. Additionally, decision making for the acquisition of 
services was left to individual units like finance, engineering, manufacturing, human 
resources, and maintenance, without top-level intervention, or expert evaluation. This 
separation of decision making hurt any chance the company had of making a coordinated 
decision across the organization. They also lacked the tools to measure the services’ 
ability to meet company needs. 

To reverse these trends, companies realized there was a need to reengineer their 
approach to services acquisition. To accomplish this company leadership would have to 
change their top-down decision-making process, to be more enterprise-centric. This 
would involve Raising commitment for a shared strategy In order to better leverage their 
buying power, and better manage service providers these companies developed systems 
to provide credible, reliable, and timely data on acquired services. Finally, the 
organization needed to develop creative ways of doing business. The following figure 
shows the four key elements of a Strategic Approach. The arrows indicate the 
complementary nature of each concept. 
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The GAO recognized that civilian companies have many of the same 
requirements for service support as DoD components. Their needs range from complex 
services such as advertising and information management, to simple services like waste 
removal and lawn care. Although they are not restricted by the same regulatory 
guidelines as the government, there is still some added benefit in understanding how they 
are able to improve their processes and procedures. 

The six companies examined in this study were chosen based upon the following 
criteria: 

• Literary research 

• University recommendations 

• industry associations 

• Research organizations 

• Other Industry experts 

The following table lists the six organizations and their functions: 
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Company 

Function 

Brunswick Corporation 

A leader ri the boating, manne engirt, titness equipment, bowling, and blNiards 
industries. 

The Dun & Bradstreet Corporatxn 

A leading provider of busv^ess credit, marketing and puxhasing inforiration. and 
receivables management services 

Electronic Data Systems (EDS) Coporation 

A leader in tne information technology seniices industry, providing busness and 
government clients nigh-value consulting, electronic business solutions, business 
process management and systems, and technology expertise. 

Exxon Mobil Corporation 

The world's largest integrated oil company, ranking first n profits, proven reserves, 
liquids produebon, natural gas productxx^, al production. arP refining capacity. 

Hasbro. Inc. 

A leader ri the design, manufacture, and marketing of toys and games, ranging from 
traditional to high-tech. 

Merrfl Lynch & Co.. Inc. 

A leading provider of investment, friancing, advisoiy. insurance, and related products 
and seni'ices. rankinq first ri U.S. and global debt arid eduity unde’wribrg. 


Table 1. Company Profiles 

The goals of their reengineering effort were to “leverage their buying power, 
reduce cost, better manage their service providers, and improve the quality of services 
provided.” Four general principles emerged as the most critical for achieving a strategic 
approach to purchasing services. They are listed below. All of the companies did not 
follow the principles exactly, but they used some variation of them. They all achieved 
the same result of “substantial” savings and service improvements. 

Comfnttmafil...Swum 4 P front oorTvnilmanr from top lM(4ars 

• ttocogni/p and crmr'untcaie l^« ur^ancy to changa sarvice ippaciong pfacticaa 

• ProvHJa limt arti lirorg anecultvo tnHiBr'.h:p mciuclmg goaM and laigali 




• OavatoD informatKXi syatem to toentrfy how much w beaig spent with which servioa provider kx what servicaa 

• Analyze the data to identify opportirwiaa to reduce coats improve servica levels, and provide better management & seance providers 


Changa... CrawSa ttvpo>tng sfrucdwa. prooMSUS. and rotos 


• Creiate or identify organizations respond ale tor coordinatvig or managing service purchases 

• isopieh proaaiva busneas ralaiionshos between and users, purchasing units, and other stakeholders 

• Implement more rlegrated 1earn.based sourcing processes 

• Create oommodity/service experts 


Support._£nabla auooaaa through sustained toadarshto. oommuracatton. and medics 

• Obtan susiarwig support from tenor laedarstvp to laoUale change 

• EstacAeT; cieer met o! commumcahon between all afrected parties 

• Demmstrata valua and creObafy o( new pro c eiiaa through uae o< metrics 

Tabic 2. Guiding Principles 

Securing Upfront Support from Top Leaders 

Senior leadership involvement from the beginning was paramount. The leaders 
provided direction, vision, and supervised the development of a common process 
approach. In some cases the leaders associated their name to new ventures in order to 
secure buy-in. 

The shift from a traditional to strategic decision-making approach required active 
participation from senior leaders. For example, traditionally, services belonged to one 
business section, while strategically they are viewed centrally. Examining how change 
occurs; traditional managers are not active participants. But, strategically, they provide 
direction and vision, by providing goals, targets, and feedback on achieving them. 
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It is important to have senior leaders spearheading reengineering efforts because 
they have the authority to direct others to support, the responsibility to set corporate 
agenda, and the power to remove barriers that block change. Previous research has 
documented that the failure of reengineering efforts are largely attributed to the lack of 
top management commitment. 

Obtain Better Knowledge on Service Spending 

The companies involved in the study conducted a spend analysis to answer two 
questions: 

• How much was being spent for services? 

• Where were the dollars going? 

The analysis revealed that they were buying similar services from numerous providers, 
and at many times varying prices. To make strategic decisions, they would have to 
develop a deliberate approach to planning and managing their acquisitions. 

Several ideas emerged from the comparison of the two approaches. First, 
traditional management focused on components and material, which did not assist in 
spending management. The strategic approach focused on developing credible, reliable, 
and timely data on services acquired. Secondly, data has traditionally been used as an 
after-the-fact reporting tool. In a strategic sense, data is used to find opportunities that 
will rationalize the supplier base and reduce costs. 

The initial step in determining spending trends is a spend analysis. A spend 
analysis should identify the following: 

• What types of services are being acquired? 

• How many suppliers for a specific service the company is using? 

• How much they are spending for that service, in total and with each 
supplier? 

• Which units within the company are purchasing the services? 

In the table below, four of the six companies shared how they capitalized on information 
learned from their spend analysis: 
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Company 

What the spend analysis revealed 

What the company did with this knowledge 

Hasbro 

Hasbro's spend analysis rovoalod mat it 
had 17 providors of temporary 
administrative, doncal. and light industrial 
personnel for 7 locations. The company 
also found that it nad inconsistent policios 
and processes, multiple contact points, and 
limited performance measures and that 
information was not being snared across 
locations. 

Hasbro consolidated its temporary personnel supplier base to a single 
provider, reduced mo number of oversight personnel, ostabiisnod a 
formal communication program, and had me contractor assume 
responsibility for developing consistent pollcios and processes. Furmor, 
by leveraging its buying povrar. Hasbro negotiated an overhead rate that 
was 45 percent lower than previously obtained and reduced its total 
spending on temporary labor from $5 million to $4.3 million. A Hasbro 
official told us its efforts often resulted in savings rangmg from 

15 percent to 45 percent and improved sen/ice levels. 

Brunswick 

Brunswick conducted a spend analysis to 
determine the types of telecommunication 
services its business units acquired, 
monthly usage rates, and cost data to help 

It dofino Its requirements and establish a 
negotiating position. It found its seven 
business units had three different 
telecommunication providors. 

Following a compietitlon in which six potential suppliers participated. 
Brunswick awarded a contract to a single supplier. Company officials told 
us their negotiated per minute usage rates wore about 60 percent loss 
man the average of its prior rates, saving Brunswick $3.2 million in the 
first 8 months of me now contract. 

Dun & 
Bradstreet 

Dun & Bradstroet’s spend analysis showed 
that the company employed hundreds of 
providors of temporary labor. While Dun & 
Bradstreet onginally planned to reduce the 
supplier base to a single provider, its 
market research found that no one 
company could moot its needs. 

Dun & Bradstreet revised its acquisition strategy to designate a preferred 
supplier mat would receive 70 percent of its business and me contract 
with a limited number of second tier suppliers that would provide 
personnel with specialized skills or provide labor to areas not son/od by 
the preferred supplier. Company officials told us mis strategy enabled 
them to acquire more actual labor per dollar since mo preferred supplier 
agreed to charge a lower profit and overhead rate man it had previously 
done when It had a smaller portion of Dun & Bradstroot's business. 

EDS 

Prior to implementing its centralized 
process for procunng information 
technology services, a spend analysis 
rovoalod mat EDS had more than 3.000 
“unlovoragod' suppliers. 

EDS initiairy conducted a strategic sourcing exercise to bettor dofino its 
needs and ostablisn a more manageable number of suppliers and 
instituted vanous interim process changes. EDS' long-term solution was 
to develop an integrated, web-based, financial and management 
information system capable of systematically matching its business 


fnanagors' noods witfi potoniial sorvico providers, capturing spending, 
wage, and overhead data by skill set and supplier, assessing supplier 
performance; and performing various financial management and 
accounting tasks. EOS officials told us that it has reduced its supplier 
base to 14 national suppliers. 6 regional suppliers, and a small set of 
providers of personnel with specialized skills, and it typically negotiates 
overhead rates that are at least one-third less than the industry average.* 
EDS officials estimated they nad saved more than $210 milion over the 
past 5 years by pursuing a more strategic approach to purchasing 
information technology senncos. 

Table 3. Spend Analysis Results 


Create Structure, Processes, and Roles to Support Enterprise Perspective 

To achieve an enterprise perspective, the companies revamped their procedures 
for acquiring services in terms of their structure, processes, and roles. Their current 
fragmented approach hindered their ability to coordinate across their organizations. They 
made three major changes: 


• Elevated and/or expanded the role of the company’s procurement 
organization. 

• Established cross-functional teams with a mix of knowledge, expertise, 
and credibility. 

• Established dedicated commodity managers to oversee key services. 
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The companies successfully performed in a proactive advisory role to their subordinates, 
instead of just being relied upon for resourcing. The role of purchasing in their 
organization changed, and those changes are shown in the following table: 


Traditional 

StFBleglc 

Independent, local organizations with 
limited VBibility ot^er the company’s total 
service sosrtdina. 

Central/matrixcd organizations rcsponsiblo 
for coordinating or managing service 
purchases. 

Reactive support role to business units. 

Proactive bus irtess rclationsnips. 

Limited coordination between business and 

Procurement process based on cross- 

purchasing units and other functions such 
as leoal or finance. 

functional teams. 

Buyers. 

Commodity/scrvicc exports. 


Table 5. Traditional v. Strategic 


Elevating Procurement Organization 

Procurement Organizations often reside deep within their business unit, 
attributing to the fragmented approach to acquiring services. The procurement 
organizations were restructured with elevated roles and given greater responsibility and 
authority. This new organization is able to assist with strategic planning, management, 
and oversight for the organization’s service spending. In the following table, Brunswick 
and EDS had informal, decentralized, independent procurement organizations, while 
ExxonMobil had none at all. All three found a way to employ this realignment to gain 
improved coordination and the optimization of resources. 


Company How the companies restructured their procurement organization 

Brunswick Historically. Brunswick had allowed its business units to operate fairly indepeodentty. with the corporation acting more 
like a holding company. According to Brunswick officials, triis decentralized structure inhibited its ability to conduct an 
onterprisowide assessment of its procurement processes and effectiveness. Brunswick’s new cnief executivo officer 
believed a more coordinated and cooperative approach would help rationalize Brunswick's supplier base and leverage 
tne company's buying power. Consequently. Brunswick established a corporate procurement organization to provide 
strategic planning for and policy and guidance to its business units and elevated the role of procurement to tne vice- 
presidential level within its corporate stmcture. 

ExxonMobil Senior corporate leadership took tne opportunity presented by the 1999 merger of Exxon and Mobil to develop a 
procurement organization that would provde the company's 11 business units a competitivo advantage by fully 
optimizing their supply chain through strategic sourcing of key servicos and gam operating cost efficiencies through a 
fully integrated procurement and payment process. The president of ExxonMobil's global services company reports 
directly to an executivo vice president. 

EDS EDS employed an informal, decentralized system for obtaining the services of programmers, systems analysts, and 

other information technology professionals employed on a temporary or contract basis. This system provided 
managers in the field almost complete authonty. but it provided EDS management with only limited visibility over the 
costs, performance, and value of the servicos acquired. In the mid-1990s, EDS faced increasing expenditures for labor 
senricos needed to meet its client needs. Additionally, in 1998. the chairman challenged the company to reduce its 
costs by S1 billion and looked for the corporate procurement organizatxxi to contnbute significantly to this goal. While 
tne procurement organization previously nad boon two to throe layers from the chief executive officer, the now 
procurement organization now reports directly to tne chief financial offcor. Additionally. EDS established a separate 
unit specifically responsible for developing, managing, and executing a comprehensive sourcing and supply process 
for acquinng information technology professionals, whch EDS officials estimated represented approximately one-third 
of tne services it acquired. 

Examples of re-structuring 


Table 6. 
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Establishing Cross-Functional Procurement Process 


Of the six companies in this study, two established Cross-Functional Procurement 
Processes to help with identifying requirements, and for evaluating potential service 
providers. The teams consisted of individuals from each business unit in order to ensure 
the right mix of knowledge, technical expertise, and credibility. The representatives 
came from various sectors like the company’s purchasing unit, internal clients, users of 
the service, and the budget office. The individuals not only worked to ensure that their 
own requirements were met, but to also assist in achieving the overall organizations 
requirements. Their tasks included the analysis of spending data, prioritizing identified 
opportunities, defining internal requirements, and conducting market research. Using this 
approach, the companies were better able to define their requirements, manage service 
providers, and meet the user’s needs at the lowest possible cost to the company. 

In the following table, ExxonMobil created a detailed sourcing handbook that was 
both practical and flexible for everyone to follow. It not only provided guidelines and 
procedures, it also provided tools, templates, and checklists for sourcing activities. They 
used a four-phase, data-driven procurement process: 

• Opportunity identification 

• Strategy development and execution 

• Supplier selection 

• Relationship management 

This process integrated their sourcing strategy into their business units’ annual 
and long term planning cycles. Taking a different approach, Hasbro used the team-based 
approach to identify annual opportunities for detailed supplier reviews and ultimate 
supplier selections. This team will evaluated the potential suppliers, conducted 
negotiations, selected the suppliers, and monitored them for future planning cycle 
feedback. 


Company How the companies use their cross-functional procurement process 

ExxonMobil ExxonMobil, the largest company we visited, established a formal procurement process. ExxonMobil officials noted 
their processes are detailed In a sourcing handbook that Is intended to be both practical and flexible enough to 
provide the basis and support for innovative sourcing strategies. The handbook provides both general guidelines and 
specific procedures and includes a suite of tools, templates, and checklists needed to effectively conduct sourcing 
activities. ExxonMobil uses a four-phase, data-driven procurement process (opportunity identification, strategy 
development, strategy execution and supplier selection, and relationship management) that involves close Interaction 
with the procurement function, internal clients, and suppliers. Sourcing strategy development is integrated within the 
business units’ annual and long term planning cycles to ensure that the strategies, priorities, and cost reduction 
objectives are fully aligned. 


Hasbro Hasbro begins Its procurement process almost a year In advance, using a team-based approach to identify 12 to 15 
opportunities each year for a more comprehensive review. According to a Hasbro officieil, about 30 people— 
representatives from purchasing, finance, logistics, and affected business units—participate in this exercise and 
consider, among other things, Hasbro's total spending for that service, the significance of the service relative to 
Hasbro's principal product lines, and market conditions. Once these opportunities are identified, smaller cross¬ 
functional teams are formed to conduct more in-depth analyses. These teams initially focused their activities on 
identifying Hasbro’s requirements and conducting market research; however, the teams subsequently obtained and 
evaluated information from potential suppliers, conducted negotiations, and finally selected the suppliers. A team 
member is generally designated to monitor the supplier, in part, as a means to provide feedback into the planning 
cycle. 
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Table 7. 


Cross-Functional Procurement Processes 


Establishing Dedicated Commodity Managers 

Three of the six companies established a full-time dedicated commodity manager 
to preside over key services. These services are either of high value or have a significant 
impact on company’s operations. ExxonMobil established eight commodity managers to 
oversee material and services. Some of their services included information technology, 
advertising, and drilling services. Brunswick established five commodity managers to 
coordinate service purchases and to serve as change agents to improve their procurement 
processes. Merrill Lynch’s commodity managers will work to define requirements with 
internal clients, negotiate with potential service providers, and will assist in resolving 
issues after a contract has been awarded. 

Enable Success through Sustained Leadership, Communication, and Metrics 

Resistance, cultural barriers, and many other impediments hindered the 
companies’ reengineering efforts. Three critical elements for overcoming these 
challenges were identified, they are: 

• Sustained leadership 

• Communication 

• Measurement 


The following table compares decision making under the traditional philosophy to the 
strategic philosophy. 


Table 8: Characteristics Promoting Successful Strategic Reengineering Efforts 

Traditional 

Strategic 

Corporate leaders not actively 
engaged in improving service 
acquisitions. 

Senior leaders actively reinforce commitment to 
achieve change. 

Business units and purchasing 
organizations do not clearly 
communicate or cooperate. 

Clear lines of communication between all 
affected parties. 

Performance measures did not exist. 

Performance measures used to demonstrate 
value and credibility of new processes. 


Source: GAO analysis. 


Sustained Leadership 

In many cases, senior leadership will offer their support in the initial stages of a 
new idea for change in service acquisitions. But, this type of effort is long-term and the 
leaders need to provide continuous support. They may even need to provide their clout in 
an effort to push for acceptance of reengineering efforts. 

Communication 
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Communication lines between business units and purchasing organizations are 
often nonexistent. This type of separation hinders the communication of requirements 
and solutions to problems. As reengineering teams work to bridge this gap, they must be 
clear in communicating their goals, rationale, and expected results if they hope to obtain 
buy-in and avoid unanticipated pitfalls. An additional success factor for reengineering 
teams is to be open minded to the needs of those affected by change, and be willing to 
alter their plans. Opening up lines of communications can potentially yield several 
benefits: 

• Timely delivery of needed services 

• Hiring of better skilled and trained consultants 

• Reduce costs 

• Provide an alternative solution to meet requirements 

Metrics 

Metrics often help with decision making because it provides credibility. The 
following is a list that shows how metrics can be used: 

• Evaluate and understand an organizations performance level 

• Identify critical processes for management attention and focus 

• Assist in obtaining knowledge for setting realistic improvement goals 

• Document results over time 

Metrics are often used to measure financial performances like total cost savings 
and cost avoidances. In this study. Dun & Bradstreet measured savings in their 
procurement process. At the beginning of the year, their senior management sets the 
target for procurements. Throughout the year, they conduct reviews to show their 
progress. 

In some cases, metrics are used to determine if internal processes are working to 
the user’s satisfaction. When companies extended the role of their procurement function, 
they used metrics to measure quality, timeliness, and how valuable the service was to the 
business unit. ExxonMobil shared employment of their extensive three-tier system to 
measure enhanced procurement functions. The top tier had eight metrics that were used 
to assess the company’s ability to meet financial, customer satisfaction, and business 
operation objectives. The second tier measures performance monitoring and 
intemal/extemal benchmarking. The third tier measured local site level daily operational 
activities. The company developed multiple metrics to bring credibility to their results. 
This is important to keep in mind when developing metrics because they help to prevent 
disagreements over numbers which could ultimately undermine the value of the process 
itself. 

Einally, some of the companies used metrics to measure compliance of a new 
process. It is important to know if new processes are being used because of the 
anticipated cost savings. In this study, EDS measured the degree to which the new 
procurement process was used in acquiring information technology systems. If their 
business units would meet the target goal by using the new system 70 to 80 percent of the 
time, then the company would save an estimated $26 mi llion. 
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Analysis: 

The data presented in this case study clearly show the benefits of using a strategic 
approach for service acquisitions. Four pillars of support were presented and the resulting 
data showed how they achieved success. The four pillars for the strategic approach are 
leader commitment, spend analysis, a supporting structure, and open lines of 
communication. 

It is important to secure Up-front Commitment from top leaders because they 
will serve as your champions. They will have access to the resources needed for success. 
Additionally, leaders want to be a part of change and if you can prove that you have a 
worthy project, then you can secure leader support. 

Having knowledge about how much is spent on services could help guide future 
decisions. The study demonstrated how cost saving could come from analyzing the data. 
This analysis could provide insight into who pays for service, who receives the service, 
and who provides the service. You could save money by streamlining the providers and 
possibly negotiate better prices. 

Enterprise concepts require supporting structures, processes, and key roles. The 
structure must allow integration of different systems. It would be helpful to establish a 
relationship between the users, purchasing units, and stakeholders. Another critical idea is 
the establishment of commodity leaders. This person could serve multiple roles. They 
could help define requirements, intervene on stalled contract negotiations, and a host of 
other tasks. 

Acquisition decision making can occur due to sustained leadership, 
communication, and metrics. It shined a light on not only the fact that senior leaders must 
be a part of any change, but that they must be the one directing the change. They can set 
goals and objectives for everyone to follow. They can open closed doors. The need for 
clear lines of communication between sections was also an area of focus. This study also 
shows the values in having credible metrics. Metrics can measure the effectiveness of 
processes, guide resourcing decisions, and provide feedback on performance. 

Conclusion: 

The study presented actionable evidence from commercial companies who took on tasks 
of change and were rewarded. In some cases, they provided explicit details on how they 
reengineered their processes and reorganized their people in an effort to achieve a 
strategic approach. Once they were able to make coordinated decisions across their 
organization, they were able to leverage resources and reduce their operating costs. 
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